Overlay of wounds based on image analysis

ABSTRACT

Embodiments consistent with the present disclosure provide systems, methods, and devices for displaying an overlay on wounds. In one example, consistent with the disclosed embodiments, an example system may: receive a real time video feed; receive image-based information associated with at least one previously captured image of a wound; generate, using the video feed and the image-based information, an overlay including an indication of a condition of the wound in the at least one previously captured image; and display, on at least one user interface, the overlay, wherein the at least one user interface is configured to display the overlay in a position associated with a position of the wound in the video feed.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is based on and claims the benefit of priority to U.S.Provisional Patent Application No. 63/133,573, filed Jan. 4, 2021, andU.S. Provisional Patent Application No. 63/195,357, filed Jun. 1, 2021,the contents of all of which are incorporated herein by reference intheir entireties.

TECHNICAL FIELD

The present disclosure relates generally to the field of imageprocessing for medical purposes. More specifically, the presentdisclosure relates to systems, methods, and computer-readable media fordisplaying an overlay on wounds.

BACKGROUND

Computer vision may be used in medical testing to determine quantitativeand qualitative clinical data. Traditionally, regulatory-approvedclinical devices use dedicated hardware such as pre-calibrated scannersthat operate in well-known and monitored capturing and illuminationconditions, together with classifiers that operate based on thecalibrated images derived by the scanners.

In recent years, smartphones have become personal mobile computers withhigh processing power, wireless Internet access, and high-resolutioncamera capabilities. However, turning a smartphone into aregulatory-approved clinical device is challenging for at least threemain reasons. First, there may be a lack of quality uniformity of thesmartphones' cameras. This can occur, for a number of reasons, includingthe fact that the settings and imaging of each brand and model ofsmartphone may differ from one to the next. Even within a particularmodel, there may be slight variations in acquired images. Second, whenusing smartphones across a host of non-uniformly lit environments, localillumination conditions can lead to varying results. Third, operation ofthe smartphone by unqualified users that may have difficulties followingstrict operation procedures.

In health administration, there is often a lack of sufficient resourcesto effectively meet healthcare demands. For example, hospitals may nothave enough medical professionals (e.g., doctors, nurses, etc.) toprovide treatment to and administer medical testing for patients. Thismay result in large inefficiencies, including ineffectively schedulingand prioritizing treatment and testing.

Accordingly, the medical field could benefit greatly from systems thatcan provide guidance to patients or other unqualified individuals (e.g.,through a user interface on a mobile device) to perform treatment and/ortesting on their own. Furthermore, it would be highly desired for suchsystems to make automatic determinations that the patient or unqualifiedindividual should not administer a given treatment and/or testing andinstead facilitate the provision of appropriate care by a qualifiedhealthcare professional.

In conventional wound care, physicians, nurses, and other health careprofessionals often must consult medical records to compare the currentcondition with previous conditions of the wound in order to makeeffective evaluations and treatment determinations, which may be timeconsuming. Moreover, health care professionals must rely on their ownevaluation of old photographs, and therefore do not gain the benefit ofcomputer vision capabilities. Accordingly, the field of wound care couldgreatly benefit from new and improved systems and methods implementingreal-time overlays on video feeds of mobile devices would provide greatbenefits in the field of wound care.

The disclosed embodiments are directed to providing new and improvedways for using personal communications devices for medical testing.

SUMMARY

Embodiments consistent with the present disclosure provide systems,methods, non-transitory computer readable media, and devices forgenerating cross section views of a wound. Conventionally, medicalpractitioners have are used to examine cross sections from at least twodifferent orthogonal orientations (for examples, in CT or MR scans).When treating wounds, however, only a frontal view of the wound isavailable to medical practitioners, given that this is the view from theposition of the camera taking the image. This makes wound treatmentespecially challenging, as the depth of the wound is an important factorin clinical decisions. Currently, the depth of the wound is estimated byeye or measured with a cotton swab, an option which is available onlywhen engaging physically with the wound, but not when examining imagesof the wound. Therefore, there is a need for systems which provide crosssection imaging and depth data using 3D reconstruction of the woundbased on data provided by a user through a standard mobile device.

One aspect of the present disclosure is directed to a non-transitorycomputer readable medium storing data and computer implementableinstructions that, when executed by at least one processor, may causethe at least one processor to perform operations for generating crosssection views of a wound. The operations may include receiving 3Dinformation of a wound based on information captured using an imagesensor associated with an image plane substantially parallel to thewound; generating a cross section view of the wound by analyzing the 3Dinformation; and providing data configured to cause a presentation ofthe generated cross section view of the wound.

Another aspect of the present disclosure is directed to a system forgenerating cross section views of a wound. The system may include amemory storing instructions; and at least one processor configured toexecute the instructions to receive 3D information of a wound based oninformation captured using an image sensor associated with an imageplane substantially parallel to the wound; generate a cross section viewof the wound by analyzing the 3D information; and provide dataconfigured to cause a presentation of the generated cross section viewof the wound.

Yet another aspect of the present disclosure is directed to acomputer-implemented method for generating cross section views of awound. The method may include receiving 3D information of a wound basedon information captured using an image sensor associated with an imageplane substantially parallel to the wound; generating a cross sectionview of the wound by analyzing the 3D information; and providing dataconfigured to cause a presentation of the generated cross section viewof the wound.

In some embodiments, the 3D information of the wound may include atleast one of a range image, a stereoscopic image, a volumetric image ora point cloud. In some embodiments, receiving the 3D information of thewound may include one or more of analyzing a video of the wound capturedusing the image sensor while the image sensor is moving, analyzing avideo of the wound depicting a motion of the wound, or analyzing atleast one image captured using the image sensor. In some embodiments,the 3D information of the wound may include at least one of a pluralityof 2D images of the wound captured from different angles, a stereoscopicimage of the wound, an image captured using an active stereo camera, oran image captured using a time-of-flight camera. In some embodiments,generating the cross section view of the wound may include selecting across section of the wound from a plurality of cross sections of thewound based on the 3D information; and generating the cross section viewof the wound by analyzing the 3D information, the cross section view ofthe wound corresponding to the selected cross section. In someembodiments, the selected cross section of the wound may correspond to adeepest point of the wound. In some embodiments, generating the crosssection view of the wound may include selecting a cross section of thewound from a plurality of cross sections of the wound based on aboundary contour of the wound; and generating the cross section view ofthe wound by analyzing the 3D information, the cross section view of thewound corresponding to the selected cross section. In some embodiments,the selected cross section of the wound may correspond to one of alongest chord of a shape of the boundary contour or a shortest chord ofthe shape of the boundary contour. In some embodiments, the selectedcross section of the wound may be perpendicular to one or more of aselected chord of a shape of the boundary contour. In some embodiments,generating a cross section view of the wound may include obtaining asegmentation of the wound based on a tissue type; selecting a crosssection of the wound of a plurality of cross sections of the wound basedon the segmentation of the wound; and generating the cross section viewof the wound by analyzing the 3D information, the cross section view ofthe wound corresponding to the selected cross section. In someembodiments, the generated cross section view of the wound may includeone or more of tissue information for at least a portion of the wound, avisual indication of a wound depth, an estimated pre-wound skin contour,or an estimated post-wound skin contour. In some embodiments, theoperations may further comprise receiving image data captured using theimage sensor; calculating a convolution of a first part of the imagedata to derive a first result value of the convolution of the first partof the image data; determining a depth of the wound at a first positionbased on the first result value; calculating a convolution of a secondpart of the image data to derive a second result value of theconvolution of the second part of the image data, the second part of theimage data differing from the first part of the image data; anddetermining a depth of the wound at a second position based on thesecond result value, the second position differing from the firstposition. In some embodiments, the generated cross section view of thewound may include a plurality of parallel cross section views of thewound. In some embodiments, the operations may further compriseestimating at least one of an original position of a skin before aformation of the wound or a future position of the skin after healing ofthe wound by analyzing the 3D information, and the provided data may bebased on at least one of the estimated original position of the skin orthe future position of the skin. In some embodiments, the provided datamay include a depth of the wound estimated based on at least one of theestimated original position of the skin or the estimated future positionof the skin. In some embodiments, the generated cross section view ofthe wound may include a visual indication of at least one of theoriginal position of the skin or the future position of the skin. Insome embodiments, at least one of estimating the original position ofthe skin or estimating the future position of the skin may includeimplementing an inpainting algorithm based on the 3D information. Insome embodiments, the wound may correspond to a first body part of apatient, the patient having a symmetrical body part to the first bodypart, and at least one of estimating the original position of the skinor estimating the future position of the skin may include receiving 3Dinformation of the symmetrical body part; and analyzing the 3Dinformation of the symmetrical body part and the 3D information of thewound.

Embodiments consistent with the present disclosure provide systems,methods, non-transitory computer readable media, and devices foranalyzing wounds using standard equipment. Conventionally, patientssuffering from wounds have to physically be attended by a medicalpractitioner to assess the evolution of their wound. Even if a patientis able to take a picture and send it to their medical practitioner,these pictures lack the quality and consistency to accurately assess theevolution of the wound. This makes wound treatment especiallychallenging, as a medical practitioner cannot accurately determine thesize, color, and shape of a wound based on pictures taken by a standardmobile communications device. Therefore, there is a need for systemswhich provide medical practitioners with consistent and calibratedimages of wounds taken with a standard mobile communications device.

One aspect of the present disclosure is directed to a non-transitorycomputer readable medium storing data and computer implementableinstructions that, when executed by at least one processor, may causethe at least one processor to perform operations for analyzing woundsusing standard equipment. The operations may include receiving one ormore images of a wound of a patient; analyzing the one or more images todetermine, based on at least a difference between values of two pixelsof the one or more images, a condition of the wound; selecting an actionbased on the determined condition of the wound; and initiating theselected action.

In some examples, an indication of a past condition of the wound at aparticular time period may be received (the particular time period maybe at least one day before the capturing of the one or more images ofthe wound, and the determination of the condition of the wound may bebased on the past condition of the wound and the analysis of the one ormore images. In some examples, the selected action may include at leastone of processing the one or more images, providing instructions to auser to capture at least one additional image of the wound, or providingparticular information associated with the condition of the wound. Insome examples, the one or more images may be analyzed to determine atleast one of a shape of the wound, a tissue composition of the wound, adepth of the wound, or a presence of an edema in a region surroundingthe wound, and wherein the determination of the condition of the woundmay be further based on the determined at least one of the shape of thewound, the tissue composition of the wound, the depth of the wound, orthe presence of the edema in the region surrounding the wound. In someexamples, the one or more images may be and/or include one or moreimages captured under artificial ultra-violet light. In some examples,the one or more images may be and/or include one or more images capturedunder artificial infrared light. In some examples, the one or moreimages may be and/or include one or more images captured using aselected physical optical filter. In some examples, the one or moreimages may include at least a first image and a second image, the firstimage may be an image captured using a first physical optical filter andthe second image may be an image captured using a second physicaloptical filter, the second physical optical filter may differ from thefirst physical optical filter, and the determination of the condition ofthe wound may be further based on an analysis of the first image and thesecond image. In some examples, the one or more images may include atleast one image depicting at least part of the wound and a calibrationelement, the calibration element may include a form of a known size, aknown shape, or a known color, and the determination of the condition ofthe wound may be based on at least one of the known size, the knownshape, or the known color. In some examples, it may be determined that aconfidence level associated with the determined condition of the woundis a first confidence level, and in response to the determination thatthe confidence level associated with the determined condition of thewound is the first confidence level, initiating the selected action maybe avoided.

In some examples, the one or more images of the wound may be and/orinclude one or more images of the wound captured using a mobilecommunications device. In one example, the mobile communications devicemay be caused to provide an instruction to a user of the mobilecommunications device to capture an image of the wound without aphysical optical filter. In one example, the mobile communicationsdevice may be caused to provide an instruction to the user to place thephysical optical filter. In one example, the mobile communicationsdevice may be caused to provide an instruction to the user to capture animage of the wound with the physical optical filter. In one example, theimage of the wound captured without the physical optical filter and theimage of the wound captured with the physical optical filter may bereceived. In one example, the image of the wound captured without thephysical optical filter and the image of the wound captured with thephysical optical filter may be analyzed to determine the condition ofthe wound. In some examples, the mobile communications device may becaused to provide an instruction to the user to place a calibrationelement in proximity to the wound (the calibration element may include aform of a known size, a known shape, and a known color), and at leastone of the known size, the known shape, or the known color may be usedin the analysis of the image of the wound captured without the physicaloptical filter and the image of the wound captured with the physicaloptical filter.

In some examples, the one or more images may be analyzed to determinethat an urgency level associated with the wound is a first level ofurgency. In one example, in response to the determination that theurgency level associated with the wound is the first level of urgency, aparticular action may be initiated. For example, the particular actionmay be configured to cause an advancement of the patient in an order oftreatment. In another example, the one or more images may include atleast a first image and a second image, the first image being an imagecaptured at least one day before a capturing of the second image, thedetermination that the urgency level associated with the wound is thefirst level of urgency may be based on a comparison of the wound in thefirst image with the wound in the second image, and the particularaction may be initiated within one hour of the capturing of the secondimage.

Another aspect of the present disclosure is directed to a kit forfacilitating capturing of medical images. The kit may include a physicaloptical filter configured to be selectively attached to a standard usermobile communications device and to manipulate light reaching a cameraincluded in the standard user mobile communications device when attachedto the standard user mobile communications device; and a calibrationelement, the calibration element including a form of a known size, aknown shape, and a known color.

In some examples, the physical optical filter may be configured toenable capturing of at least two medical images of a wound by the cameraincluded in the standard user mobile communications device, includingcapturing at least one image using the physical optical filter andcapturing at least one image without the physical optical filter, andthe calibration element may be configured to enable color calibration ofthe at least one image captured using the physical optical filter and toenable calibration of the at least one image captured without thephysical optical filter. In some examples, the physical optical filtermay be shaped to envelop at least one corner of the standard user mobilecommunications device while covering the camera included in the standarduser mobile communications device. In some examples, the physicaloptical filter may include an adhesive side configured to attach thephysical optical filter to the standard user mobile communicationsdevice.

Yet another aspect of the present disclosure is directed to acomputer-implemented method for analyzing wounds using standardequipment. The method may include receiving one or more images of awound of a patient; analyzing the one or more images to determine, basedon at least a difference between values of two pixels of the one or moreimages, a condition of the wound; selecting an action based on thedetermined condition of the wound; and initiating the selected action.

Embodiments consistent with the present disclosure provide systems,methods, non-transitory computer readable media, and devices forgenerating visual time series of wounds. Conventionally, in each checkupduring the treatment of a wound, images of the wound are taken with amobile device held by a user's hand. The images are taken from differentorientations of the camera with respect to the wound, in differingilluminations, and sometimes even with different cameras with differentcapturing parameters. This makes creating a visual time series view of awound especially challenging as images of the same wound throughout itstreatment may vary greatly and analysis of the wound based on the imagesmay prove flawed because of the images not being normalized. Therefore,there is a need to create a visual time series view of the progressionof the wound where the viewing angle, illumination, colors, distance,and other appropriate characteristics of the images are normalized.

One aspect of the present disclosure is directed to a non-transitorycomputer readable medium storing data and computer implementableinstructions that, when executed by at least one processor, may causethe at least one processor to perform operations for generating visualtime series of wounds. The operations may include receiving at least afirst image data record and a second image data record, the first imagedata record corresponding to a first point in time and including a firstone or more images of a wound captured at the first point in time, andthe second image data record corresponding to a second point in time andincluding a second one or more images of the wound captured at thesecond point in time; obtaining an image of the wound from a particularpoint of view corresponding to the first point in time by analyzing thefirst image data record; generating a simulated image of the wound fromthe particular point of view corresponding to the second point in timeby analyzing the second image data record, wherein the second one ormore images of the wound do not include an image of the wound from theparticular point of view; and generating a visual time series view ofthe wound including at least the image of the wound from the particularpoint of view corresponding to the first point in time and the simulatedimage of the wound from the particular point of view corresponding tothe second point in time.

Another aspect of the present disclosure is directed to acomputer-implemented method for generating visual time series views ofwounds. The method may include receiving a first image data record and asecond image data record, the first image data record corresponding to afirst point in time and including a first one or more images of a woundcaptured at the first point in time, and the second image data recordcorresponding to a second point in time and including a second one ormore images of the wound captured at the second point in time; obtainingan image of the wound from a particular point of view corresponding tothe first point in time by analyzing the first image data record;generating a simulated image of the wound from the particular point ofview corresponding to the second point in time by analyzing the secondimage data record, wherein the second one or more images of the wound donot include an image of the wound from the particular point of view; andgenerating a visual time series view of the wound including at least theimage of the wound from the particular point of view corresponding tothe first point in time and the simulated image of the wound from theparticular point of view corresponding to the second point in time.

Yet another aspect of the present disclosure is directed to a system forgenerating visual time series of wounds. The system may include a memorystoring instructions; and at least one processor configured to executethe instructions to receive at least a first image data record and asecond image data record, the first image data record corresponding to afirst point in time and including a first one or more images of a woundcaptured at the first point in time, and the second image data recordcorresponding to a second point in time and including a second one ormore images of the wound captured at the second point in time; obtain animage of the wound from a particular point of view corresponding to thefirst point in time by analyzing the first image data record; generate asimulated image of the wound from the particular point of viewcorresponding to the second point in time by analyzing the second imagedata record, wherein the second one or more images of the wound do notinclude an image of the wound from the particular point of view; andgenerate a visual time series view of the wound including at least theimage of the wound from the particular point of view corresponding tothe first point in time and the simulated image of the wound from theparticular point of view corresponding to the second point in time.

In some examples, the image of the wound from the particular point ofview corresponding to the first point in time may be a simulated imageof the wound based on the first image data record. In some examples, theimage of the wound from the particular point of view corresponding tothe first point in time may be an image of the first one or more imagesof the wound. In some examples, the second image data record may includemotion data captured using an accelerometer associated with an imagesensor used to capture the second one or more images of the wound, andthe analyzing the second image data record may include analyzing themotion data. In some examples, generating the simulated image of thewound from the particular point of view corresponding to the secondpoint in time may include generating the simulated image to haveselected illumination characteristics. In one example, generating thesimulated image of the wound from the particular point of viewcorresponding to the second point in time may further include analyzingthe image of the wound from the particular point of view correspondingto the first point in time to select the selected illuminationcharacteristics. In some examples, the images of the wound from theparticular point of view corresponding to the first point in time and tothe second point in time may both correspond to a same treatment phaseof a treatment cycle of the wound. In one example, generating thesimulated image of the wound from the particular point of viewcorresponding to the second point in time may further include analyzingthe image of the wound from the particular point of view correspondingto the first point in time to determine a treatment phase of thetreatment cycle of the wound corresponding to the image of the woundfrom the particular point of view corresponding to the first point intime, and generating the simulated image of the wound from theparticular point of view corresponding to the second point in time tocorrespond to the determined treatment phase. In some examples, eachimage of the images in the visual time series view of the wound maycorrespond to a point in time, and the images in the visual time seriesview of the wound may be ordered based on the corresponding points intime. In some examples, the images of the wound from the particularpoint of view corresponding to the first point in time and to the secondpoint in time may both correspond to a same distance from the wound. Forexample, generating the simulated image of the wound from the particularpoint of view corresponding to the second point in time may includegenerating the simulated image of the wound from the particular point ofview corresponding to the second point in time by causing a distancefrom the wound in the simulated image to be equal to the distance fromthe wound associated with the image of the wound from the particularpoint of view corresponding to the first point in time. In someexamples, the images of the wound from the particular point of viewcorresponding to the first point in time and to the second point in timemay both have a same spatial orientation. For example, generating thesimulated image of the wound from the particular point of viewcorresponding to the second point in time may include generating thesimulated image of the wound from the particular point of viewcorresponding to the second point in time by causing a spatialorientation in the simulated image to be equal to a spatial orientationassociated with the image of the wound from the particular point of viewcorresponding to the first point in time. In some examples, pixels of atleast one matching pair of pixels of the image of the wound from theparticular point of view corresponding to the first point in time andfrom the simulated image of the wound from the particular point of viewcorresponding to the second point in time may correspond to a samephysical length. In some examples, a convolution of a part of an imageof the first one or more images may be calculated to derive a firstresult value, a convolution of a part of an image of the second one ormore images may be calculated to derive a second result value, and avalue of at least one pixel of the simulated image of the wound from theparticular point of view corresponding to the second point in time maybe determined based on the first result value and the second resultvalue. In some examples, a first image of the first one or more imagesmay be analyzed to detect a region of the wound corresponding to aparticular tissue type in the first image, a second image of the secondone or more images may be analyzed to detect a region of the woundcorresponding to the particular tissue type in the second image, and avalue of at least one pixel of the simulated image of the wound from theparticular point of view corresponding to the second point in time maybe determined based on the region of the wound corresponding to theparticular tissue type in the first image and the region of the woundcorresponding to the particular tissue type in the second image. In someexamples, each particular image of the wound from the particular pointof view corresponding to the first point in time and to the second pointin time may include a visual indicator of a region of the woundcorresponding to a particular tissue type in the particular image. Insome examples, each particular image of the wound from the particularpoint of view corresponding to the first point in time and to the secondpoint in time may include a visual indicator of a depth of the wound ata particular location.

Embodiments consistent with the present disclosure provide systems,methods, and computer readable media for rearranging and selectingframes of medical videos. One embodiment consistent with the presentdisclosure may include a non-transitory computer readable medium storingdata and computer implementable instructions that, when executed by atleast one processor, cause the at least one processor to performoperations for rearranging and selecting frames of a medical video. Theoperations may include: obtaining a desired property of a simulatedtrajectory of a virtual camera; receiving a first video of a woundcaptured by a moving camera, the first video including a plurality offrames; using the desired property of the simulated trajectory of thevirtual camera to analyze the first video to select at least two framesof the plurality of frames corresponding to the simulated trajectory ofthe virtual camera; using the desired property of the simulatedtrajectory of the virtual camera to select an order for the selected atleast two frames; and rearranging the at least two frames based on theselected order to create a new video of the wound that represents thesimulated trajectory of the virtual camera.

According to another embodiment of the present disclosure, a system forrearranging and selecting frames of a medical video may be provided. Thesystem may include a memory storing instructions; and at least oneprocessor configured to execute the instructions to: obtain a desiredproperty of a simulated trajectory of a virtual camera; receive a firstvideo of a wound captured by a moving camera, the first video includinga plurality of frames; use the desired property of the simulatedtrajectory of the virtual camera to analyze the first video to select atleast two frames of the plurality of frames corresponding to thesimulated trajectory of the virtual camera; use the desired property ofthe simulated trajectory of the virtual camera to select an order forthe selected at least two frames; and rearrange the at least two framesbased on the selected order to create a new video of the wound thatrepresents the simulated trajectory of the virtual camera.

According to yet another embodiment of the present disclosure, acomputer-implemented method for rearranging and selecting frames of amedical video may be provided. The method may include: obtaining adesired property of a simulated trajectory of a virtual camera;receiving a first video of a wound captured by a moving camera, thefirst video including a plurality of frames; using the desired propertyof the simulated trajectory of the virtual camera to analyze the firstvideo to select at least two frames of the plurality of framescorresponding to the simulated trajectory of the virtual camera; usingthe desired property of the simulated trajectory of the virtual camerato select an order for the selected at least two frames; and rearrangingthe at least two frames based on the selected order to create a newvideo of the wound that represents the simulated trajectory of thevirtual camera.

In some examples, a trajectory of the moving camera may be a pathfollowed by the moving camera from a first position to a secondposition, and the simulated trajectory may be a generated path betweenthe first position and the second position. For example, the trajectoryof the moving camera may include a diversion rendering at least aportion of the trajectory non-linear, and in one example the simulatedtrajectory does not include the diversion. In some examples, creatingthe new video may comprise generating at least one synthetic frame byanalyzing the first video, and wherein the new video includes the atleast one synthetic frame. In some examples, the simulated trajectorymay be selected based on a second video of the wound captured at adifferent time. For example, data configured to cause a display of thesecond video may be provided in conjunction with a display of the newvideo. In some examples, the simulated trajectory may be a standardwound viewing trajectory. In some examples, at least a portion of thesimulated trajectory may be selected to be substantially on an arc of acircle, and the wound may be located at or near the center of thecircle. In some examples, the desired property of the simulatedtrajectory of the virtual camera may include. a desired moving directionof the virtual camera. For example, obtaining the desired property ofthe simulated trajectory may comprise selecting the desired movingdirection of the virtual camera based on a contour of the wound. In someexamples, the desired property of the simulated trajectory of thevirtual camera may include a desired velocity of the virtual camera. Insome examples, the desired property of the simulated trajectory of thevirtual camera may include a desired distance of the virtual camera fromthe wound. In some examples, at least one image of the wound may beanalyzed to determine a condition of at least part of the wound, and thesimulated trajectory of the virtual camera may be determined based onthe condition of the at least part of the wound. In some examples, atleast one image of the wound may be analyzed to identify a first regionof the wound corresponding to a first tissue type and a second region ofthe wound corresponding to a second tissue type, and the simulatedtrajectory of the virtual camera may be determined based on a dimensionof the first region of the wound, the first tissue type, a dimension ofthe second region of the wound, and the second tissue type.

In some examples, a first correction factor associated with a firstportion of the new video of the wound and a second correction factorassociated with a second portion of the new video of the wound may bereceived. In some examples, creating the new video of the wound mayinclude modifying frames of the first portion of the new video of thewound based on the first correction factor and modifying frames of thesecond portion of the new video of the wound based on the secondcorrection factor. In one example, the first correction factor maycorrespond to a first illumination condition and the second correctionfactor may correspond to a second illumination condition. In oneexample, the first correction factor may correspond to a first distancefrom the wound and the second correction factor may correspond to asecond distance from the wound. In one example, at least one image ofthe wound may be analyzed to identify a first region of the woundcorresponding to a first tissue type and a second region of the woundcorresponding to a second tissue type. The first portion of the newvideo of the wound may be determined based on the first region of thewound and the second portion of the new video of the wound may bedetermined based on the second region of the wound. Further, the firstcorrection factor may be determined based on the first tissue type andthe second correction factor may be determined based on the secondtissue type.

Embodiments consistent with the present disclosure provide systems,methods, and devices for providing guidance for capturing images ofwounds. Conventionally, physicians are limited in their ability toaccurately analyze a wound's condition and determine appropriatetreatment when presented with images of the wound. Even in circumstanceswhere physicians have the opportunity to inspect wounds in person, theyare not equipped to perform an analysis of the present wound at the sameefficacy as a computerized system configured to analyze a wound ifprovided with a comparable amount of visual data (e.g., by generatingaccurate three-dimensional models and/or measurements of the wound andcorrelating such models with diagnostic data). Accordingly, there is aneed for systems and methods for providing guidance for imaging woundsto provide physicians and computerized systems with sufficient data tomake effective diagnostic and treatment determinations based on thecondition of a patient's wound. Moreover, while a skilled practitionermay know how to capture wound images in a medically beneficial way,image capturing of a wound by medical practitioners limits the capturingto events where the patient and the practitioner meets, such as homevisits or clinic visits. However, providing appropriate guidance mayenable the patient, or any caregiver of the patient, to capture thewound images at higher frequencies. Through remote medicine, or throughautomatic analysis of the wound images, the higher frequency ofcapturing may translate for higher frequency of monitoring. Especiallyfor non-skilled user, providing the guidance in real time when the woundimages are captured, and adjusting the guidance to the actions of theuser or the status of the wound in real time, may be preferred tooffline training that prepares the user to react to differentsituations.

In one example, consistent with the disclosed embodiments, an examplesystem may: receive a plurality of frames from at least one image sensorassociated with a mobile device, at least one of the plurality of framescontaining an image of a wound; display, on the mobile device, a realtime video including at least a portion of the plurality of frames and avisual overlay indicating a desired position of the wound; detect, basedon at least part of the plurality of frames, that the wound is in thedesired position; when the wound is in the desired position, display anindication on the mobile device to move the mobile device in a desireddirection; receive motion data from at least one motion sensorassociated with the mobile device; detect, based on the motion data ofthe mobile device, that the mobile device has moved in the desireddirection; and, when the mobile device has moved in the desireddirection, display an additional indication on the mobile device.

In some embodiments, the visual overlay may include an indication of adesired position for a center of the wound. In some embodiments, thevisual overlay may include an indication of a bounding shape for thewound in the video. In some embodiments, the operations may furtherinclude calculating a convolution of the at least part of the pluralityof frames to derive a result value of the calculated convolution;determining an actual position of the wound based on the derived resultvalue of the calculated convolution; and comparing the actual positionof the wound with the desired position of the wound to detect that thewound is in the desired position. In some embodiments, the operationsmay further include detecting, based on an analysis of the at least oneframe of the plurality of frames, that the wound is not in the desiredposition for at least a specified period of time; and when the wound isnot in the desired position for at least the specified period of time,displaying, on the mobile device, an indication to correct an actualposition of the wound in the video. In some embodiments, the operationsmay further include before detecting that the wound is in the desiredposition, displaying, on the mobile device, an indication to correct anactual position of the wound in the video; and after detecting that thewound is in the desired position, halting the display of the indicationto correct the actual position of the wound in the video. In someembodiments, the additional indication may include an instruction tomove the mobile device in a different direction. In some embodiments,the operations may further include detecting, based on an analysis ofthe motion data of the mobile device, that the mobile device has movedin a direction different from the desired direction; and when the mobiledevice has moved in the direction different from the desired direction,displaying an indication on the mobile device to correct the movement ofthe mobile device. In some embodiments, the operations may furtherinclude detecting, based on an analysis of at least one frame of theplurality of frames, that illumination conditions are not satisfactory;and when the illumination conditions are not satisfactory, displaying anindication on the mobile device to take an action to improve theillumination conditions. In some embodiments, the operations may furtherinclude generating a user rating based on an analysis of at least oneframe of the plurality of frames. In some embodiments, the operationsmay further include detecting, based on an analysis of at least oneframe of the plurality of frames, the presence of a shadow in theplurality of frames; detecting that the shadow is cast over the wound inthe plurality of frames; and determining, based on an analysis of theshadow in the plurality of frames, information related to an objectcasting the shadow. In some embodiments, the operations may furtherinclude determining, based on the information, a particular action; andwhen the shadow is cast over the wound, causing a performance of theparticular action. In some embodiments, causing the performance of theparticular action may include displaying an indication on the mobiledevice to move the mobile device to a different location. In someembodiments, the particular action may include activating a flashfeature associated with the mobile device. In some embodiments, causingthe performance of the particular action may include displaying anindication on the mobile device to move the object casting the shadow sothat it no longer casts a shadow on the wound. In some embodiments, theinformation may include an indication that the object casting the shadowis the mobile device. In some embodiments, the information may includean indication that the object casting the shadow is a hand holding themobile device. In some embodiments, the operations may further comprisemodifying, based on the information and when the shadow is cast over thewound, at least one parameter associated with the at least one imagesensor.

Embodiments consistent with the present disclosure provide systems,methods, and devices for providing wound capturing guidance. In oneexample, consistent with the disclosed embodiments, an example systemmay: display, on a mobile device, a user interface configured to guide apatient through one or more steps for performing a medical action, theplurality of steps including at least: using at least one item of amedical kit; and capturing at least one image of at least part of the atleast one item of the medical kit using at least one image sensorassociated with the mobile device. The example system may also: detect afailure to successfully complete the medical action; select from one ormore alternative reactions, a reaction to the detected failure likely tobring a successful completion of the medical action; and provideinstructions associated with the selected reaction.

In some embodiments, the one or more alternative reactions may includeat least two of triggering a provision of an additional medical kit tothe patient; triggering an approach to the patient by a person; ortriggering a provision of additional guidance to the patient using theuser interface. In some embodiments, the selected reaction may includetriggering a provision of an additional medical kit to the patient, andthe provided instructions may be configured to cause the provision ofthe additional medical kit to the patient. In some embodiments, theselected reaction may include triggering an approach to the patient by aperson, and the provided instructions may be configured to alert theperson to approach the patient. In some embodiments, the selectedreaction may include triggering a provision of additional guidance tothe patient, and the provided instructions may be configured to provideadditional guidance to the patient using the user interface. In someembodiments, the selection of the reaction may be based on a type of thefailure detected. In some embodiments, the selection of the reaction maybe based on a result of the detected failure. In some embodiments,detecting a failure may include identifying the one or more failed stepsfor performing a medical action and selecting a reaction is based on theone or more failed steps identified. In some embodiments, the steps forperforming a medical action may further include at least one ofpositioning a calibrator sticker; positioning a dipstick adjacent to acalibrator; dipping a dipstick in a medical sample; or blotting adipstick. In some embodiments, detecting a failure may further includeat least one of detecting that the calibrator sticker is incorrectlypositioned, the dipstick is incorrectly positioned adjacent to thecalibrator, the dipstick is improperly dipped in the medical sample, orthe dipstick is improperly blotted. In some embodiments, the at leastone item of the medical kit may be at least one of a dipstick; or acalibrator. In some embodiments, the detected failure may include afailure to perform a physical action using the at least one item of themedical kit. In some embodiments, the detected failure may include afailure to capture the at least one image within a particular timewindow. In some embodiments, detecting the failure may be based on ananalysis of the at least one captured image. In some embodiments,detecting the failure may include detecting that the user interface wasshut down before completing at least one of the steps for performing amedical action. In some embodiments, the operations may further includedetermining that the failure necessitates a usage of an alternative itemto the at least one item of the medical kit for a successful completionof the medical action; when the medical kit includes the alternativeitem, the selected reaction includes at least one of triggering anapproach to the patient by a person or triggering a provision ofadditional guidance to the patient using the user interface; and whenthe medical kit does not include the alternative item, the selectedreaction includes at least one of triggering a provision of anadditional medical kit to the patient or triggering a performance of themedical action by a medical professional. In some embodiments, theoperations may further include determining that the failure necessitatesa usage of an alternative item to the at least one item of the medicalkit for a successful completion of the medical action; determining thatthe patient has a first characteristic; in response to the patienthaving the first characteristic, triggering a provision of an additionalmedical kit to the patient; determining that the patient has a secondcharacteristic; and in response to the patient having the secondcharacteristic, triggering a performance of the medical action by amedical professional. In some embodiments, the operations may furtherinclude determining that the failure does not necessitate a usage of analternative item to the at least one item of the medical kit for asuccessful completion of the medical action; determining that thepatient has a first characteristic; in response to the patient havingthe first characteristic, triggering an approach to the patient by aperson; determining that the patient has a second characteristic; and inresponse to the patient having the second characteristic, triggering aprovision of additional guidance to the patient using the userinterface.

Embodiments consistent with the present disclosure provide systems,methods, and devices for displaying an overlay on wounds. In oneexample, consistent with the disclosed embodiments, an example method orsystem may: receive a real time video feed; receive image-basedinformation associated with at least one previously captured image of awound; generate, using the video feed and the image-based information,an overlay including an indication of a condition of the wound in the atleast one previously captured image; and display, on at least one userinterface, the overlay, wherein the at least one user interface isconfigured to display the overlay in a position associated with aposition of the wound in the video feed.

In some examples, the at least one previously captured image may becaptured at least one day before the video feed is captured. In someexamples, the video feed may include a plurality of wounds, and thewound may be selected from the plurality of wounds. In some examples,the overlay may be displayed on the user interface feed in real time. Insome examples, the indication of the condition of the wound may includea visual indication of a contour of the wound in the at least onepreviously captured image. In some examples, the indication of thecondition of the wound may include an indication of at least onemeasurement of the wound in the at least one previously captured image(for example, the at least one measurement may include at least one of alength, an area, a volume, or a depth of the wound). In some examples,the indication of the condition of the wound may include a visualindication of a segment of the wound in the at least one previouslycaptured image corresponding to a tissue type. In some examples, theindication of the condition of the wound may include a visual indicationof a color of a portion of the wound in the at least one previouslycaptured image. In some examples, the indication of the condition of thewound may include a visual indication of a severity of the wound in theat least one previously captured image.

In some examples, receiving the image-based information may compriseaccessing a plurality of records, each record of the plurality ofrecords corresponding to a different wound, selecting a recordcorresponding to the wound of the plurality of records based on thevideo feed, and obtaining the image-based information from the selectedrecord.

In some examples, the overlay may be displayed in real time using atransparent optical system included in a wearable device, the real timevideo feed may be a real time video feed captured using an image sensorincluded in the wearable device, the wound may be visible to a userwearing the wearable device through the transparent optical system, andthe display of the overlay may be configured to make the overlay appearto the user wearing the wearable device at least partly over the wound.

In some examples, the at least one user interface may be associated witha mobile device. For example, the at least one user interface may beconfigured to automatically adjust the position of the displayed overlaybased on detected movement of the mobile device.

In some examples, second image-based information associated with asecond at least one previously captured image of the wound may bereceived. Further, a second indication may be included in the overlay.The second indication may be an indication of a condition of the woundin the second at least one previously captured image. The condition ofthe wound in the second at least one previously captured image maydiffer from the condition of the wound in the at least one previouslycaptured image. In one example, the overlay may further include anindication of a capturing time associated with the at least onepreviously captured image and an indication of a capturing timeassociated with the second at least one previously captured image. Inanother example, the condition of the wound in the at least onepreviously captured image may correspond to a first point in time, andthe condition of the wound in the second at least one previouslycaptured image may correspond to a second point in time. Further, theimage-based information and the second image-based information may beused to determine a condition of the wound corresponding to a thirdpoint in time (the third point in time may differ from the first pointin time and the second point in time), and a third indication may beincluded in the overlay. The third indication may be an indication of acondition of the wound corresponding to the third point in time.

In some examples, a convolution of at least part of the at least onepreviously captured image may be calculated to derive a result value. Inresponse to a first result value, a first indication of the condition ofthe wound in the at least one previously captured image may be includedin the overlay, and in response to a second result value, a secondindication of the condition of the wound in the at least one previouslycaptured image may be included in the overlay. The second indication maydiffer from the first indication.

Embodiments may include a display, on a mobile device, and a userinterface configured to guide a patient through a plurality of steps forperforming a medical action, the plurality of steps including at least:using at least one item of a medical kit; and capturing at least oneimage of at least part of the at least one item of the medical kit usingat least one image sensor associated with the mobile device. The examplesystem may also: detect a failure to successfully complete the medicalaction; select from a plurality of alternative reactions, a reaction tothe detected failure likely to bring a successful completion of themedical action; and provide instructions associated with the selectedreaction.

In some embodiments, the at least one previously captured image may becaptured at least one day before the video feed is captured. In someembodiments, the video feed may include a plurality of wounds, and theoperations may further include selecting the wound from the plurality ofwounds. In some embodiments, receiving the image-based information mayinclude accessing a plurality of records, each record of the pluralityof records corresponding to a different wound; selecting a recordcorresponding to the wound of the plurality of records based on thevideo feed; and obtaining the image-based information from the selectedrecord. In some embodiments, the overlay may be displayed on the userinterface feed in real time. In some embodiments, the overlay may bedisplayed in real time using a transparent optical system included in awearable device, the real time video feed being a real time video feedcaptured using an image sensor included in the wearable device, thewound being visible to a user wearing the wearable device through thetransparent optical system, and the display of the overlay beingconfigured to make the overlay appear to the user wearing the wearabledevice at least partly over the wound. In some embodiments, the at leastone user interface may be associated with a mobile device. In someembodiments, the at least one user interface may be configured toautomatically adjust the position of the displayed overlay based ondetected movement of the mobile device. In some embodiments, theoperations may further include receiving second image-based informationassociated with a second at least one previously captured image of thewound; and including a second indication in the overlay, the secondindication being an indication of a condition of the wound in the secondat least one previously captured image, the condition of the wound inthe second at least one previously captured image differing from thecondition of the wound in the at least one previously captured image. Insome embodiments, the overlay may further an indication of a capturingtime associated with the at least one previously captured image and anindication of a capturing time associated with the second at least onepreviously captured image. In some embodiments, the condition of thewound in the at least one previously captured image may correspond to afirst point in time, the condition of the wound in the second at leastone previously captured image may correspond to a second point in time,and the operations may further include using the image-based informationand the second image-based information to determine a condition of thewound corresponding to a third point in time, the third point in timediffering from the first point in time and the second point in time; andincluding a third indication in the overlay, the third indication beingan indication of a condition of the wound corresponding to the thirdpoint in time. In some embodiments, the operations may further includecalculating a convolution of at least part of the at least onepreviously captured image to derive a result value; in response to afirst result value, including in the overlay a first indication of thecondition of the wound in the at least one previously captured image;and in response to a second result value, including in the overlay asecond indication of the condition of the wound in the at least onepreviously captured image, the second indication differing from thefirst indication. In some embodiments, the indication of the conditionof the wound may include a visual indication of a contour of the woundin the at least one previously captured image. In some embodiments, theindication of the condition of the wound may include an indication of atleast one measurement of the wound in the at least one previouslycaptured image. In some embodiments, the at least one measurement mayinclude at least one of a length, an area, a volume, or a depth of thewound. In some embodiments, the indication of the condition of the woundmay include a visual indication of a segment of the wound in the atleast one previously captured image corresponding to a tissue type. Insome embodiments, the indication of the condition of the wound mayinclude a visual indication of a color of a portion of the wound in theat least one previously captured image. In some embodiments, theindication of the condition of the wound may include a visual indicationof a severity of the wound in the at least one previously capturedimage.

Consistent with other disclosed embodiments, non-transitorycomputer-readable storage media may store program instructions, whichare executed by at least one processing device, and may perform any ofthe methods described herein.

The foregoing general description and the following detailed descriptionare for example and explanatory only and are not restrictive of theclaims.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A is a schematic illustration of an example system that uses imagedata captured by mobile communications devices for medical testing,consistent with some embodiments of the present disclosure.

FIG. 1B is a flowchart of an example process for completing a medicalexamination, consistent with some embodiments the present disclosure.

FIG. 1C is an example flow diagram illustrating communications exchangesbetween different entities implementing the process of FIG. 1B,consistent with some embodiments of the present disclosure.

FIG. 2 is a block diagram illustrating some of the components of thesystem of FIG. 1A, consistent with some embodiments of the presentdisclosure.

FIG. 3 is a schematic illustration of how two different mobilecommunications devices can obtain the same test results, consistent withsome embodiments of the present disclosure.

FIG. 4A is an illustration of one aspect of the disclosure where theexamined object is a tissue feature, consistent with some embodiments ofthe present disclosure.

FIG. 4B is an illustration of another aspect of the disclosure where theexamined object is a dipstick, consistent with some embodiments of thepresent disclosure.

FIG. 5A is an illustration of a mobile communications device capturingan image or video of an example wound, the device parallel to the wound,consistent with some embodiments of the present disclosure.

FIG. 5B is an illustration of a mobile communications device capturingan image or video of an example wound, the device substantially parallelto the wound, consistent with some embodiments of the presentdisclosure.

FIG. 6 is an illustration of a mobile communications device capturingone or more images or videos of an example wound at different angleswhile remaining parallel to the wound, consistent with some embodimentsof the present disclosure.

FIG. 7 is an illustration of an example cross section view of a wound,consistent with some embodiments of the present disclosure.

FIG. 8A is an illustration of an example of a first cross section of awound, consistent with some embodiments of the present disclosure.

FIG. 8B is an illustration of an example of a second cross section of awound, consistent with some embodiments of the present disclosure.

FIG. 8C is an illustration of an example of a third cross section of awound, consistent with some embodiments of the present disclosure.

FIG. 9 is an illustration of a front view of a wound, consistent withsome embodiments of the present disclosure.

FIG. 10 is an illustration of a front view of a wound segmented bytissue type, consistent with some embodiments of the present disclosure.

FIG. 11 is a flowchart of an example process for generating crosssection views of a wound, consistent with some embodiments of thepresent disclosure.

FIG. 12 is an illustration of a mobile communications device capturingan image, consistent with some embodiments of the present disclosure.

FIG. 13A is an illustration of an infected wound segmented into pixels,consistent with some embodiments of the present disclosure.

FIG. 13B is an illustration of a healthy wound segmented into pixels,consistent with some embodiments of the present disclosure.

FIG. 14 is an illustration of an infected wound with a calibrationelement, consistent with some embodiments of the present disclosure.

FIG. 15 is an illustration of a physical optical filter affixed to astandard mobile communications device, consistent with some embodimentsof the present disclosure.

FIG. 16 is an illustration of a kit including a physical optical filterand a calibration element, consistent with some embodiments of thepresent disclosure.

FIG. 17 is a flowchart of an example process for analyzing wounds usingstandard user equipment, consistent with some embodiments of the presentdisclosure.

FIG. 18 is an illustration of a mobile communications device capturingan image or video of an example wound, consistent with some embodimentsof the present disclosure.

FIG. 19 is an illustration of an image data record corresponding to afirst point in time, consistent with some embodiments of the presentdisclosure.

FIG. 20 is an illustration of an image data record corresponding to asecond point in time, consistent with some embodiments of the presentdisclosure.

FIG. 21 is an illustration of a simulated image data recordcorresponding to a second point in time, consistent with someembodiments of the present disclosure.

FIG. 22 is an illustration of a pixelated image and simulated image,consistent with some embodiments of the present disclosure.

FIG. 23 is an illustration of visual time series views of a wound,consistent with some embodiments of the present disclosure.

FIG. 24 is an example process for generating visual time series views ofwounds, consistent with some embodiments of the present disclosure.

FIG. 25A is an illustrative X-Y view of an example simulated trajectoryof a virtual camera for creating a new video of a wound of a patient,consistent with some embodiments of the present disclosure.

FIG. 25B is an illustrative Y-Z view of an example simulated trajectoryof a virtual camera for creating a new video of a wound of a patient,consistent with some embodiments of the present disclosure.

FIG. 25C is another illustrative Y-Z view of an example virtual cameramoving along a simulated trajectory for creating a new video of a woundof a patient, consistent with some embodiments of the presentdisclosure.

FIG. 26 is a flowchart of an example process for rearranging andselecting frames of medical videos, consistent with some embodiments ofthe present disclosure.

FIG. 27 is an illustration of a mobile communications device capturingan image of an example wound on the arm of a patient, consistent withsome embodiments of the present disclosure.

FIG. 28A is an illustration of a mobile communications device displayingan example overlay on an image of a wound, consistent with someembodiments of the present disclosure.

FIG. 28B is another illustration of a mobile communications devicedisplaying an example overlay on an image of a wound, consistent withsome embodiments of the present disclosure.

FIG. 29 is an illustration of a mobile communications device capturing aseries of images in different positions of an example wound on the armof a patient, consistent with some embodiments of the presentdisclosure.

FIG. 30 is an illustration of a mobile communications device capturingan image of where a shadow is being cast over a wound of a patient,consistent with some embodiments of the present disclosure.

FIG. 31 is a flowchart of an example process for providing woundcapturing guidance, consistent with some embodiments of the presentdisclosure.

FIG. 32 is an illustration of a mobile device with a user interface forguiding a user through a series of steps in a medical image capturingapplication, consistent with some embodiments of the present disclosure.

FIG. 33 is a flowchart of an example process for selective reaction to afailure to successfully complete a medical action using a medical imagecapturing application, consistent with some embodiments of the presentdisclosure.

FIG. 34 is an illustration of an example mobile device configured todisplay an overlay on one or more wounds in a video feed, consistentwith some embodiments of the present disclosure.

FIG. 35 is an illustration of another example device configured todisplay an overlay on one or more wounds in a video feed, consistentwith some embodiments of the present disclosure.

FIG. 36 is a flowchart of an example process for displaying an overlayon one or more wounds in a video feed, consistent with some embodimentsof the present disclosure.

DETAILED DESCRIPTION

The following detailed description refers to the accompanying drawings.Wherever possible, the same reference numbers are used in the drawingsand the following description to refer to the same or similar parts.While several illustrative embodiments are described herein,modifications, adaptations and other implementations are possible. Forexample, substitutions, additions, or modifications may be made to thecomponents illustrated in the drawings, and the illustrative methodsdescribed herein may be modified by substituting, reordering, removing,or adding steps to the disclosed methods. Accordingly, the followingdetailed description is not limited to the disclosed embodiments andexamples, but is inclusive of general principles described herein inaddition to the general principles encompassed by the appended claims.

The present disclosure is directed to systems and methods for processingimages captured by an image sensor. As used herein, the term “imagesensor” refers to any device capable of detecting and converting opticalsignals in the near-infrared, infrared, visible, and ultravioletspectrums into electrical signals. Examples of image sensors may includedigital cameras, phone cameras, semiconductor charge-coupled devices(CCD), active pixel sensors in complementary metal-oxide-semiconductor(CMOS), or N-type metal-oxide-semiconductor (NMOS, Live MOS). Theelectrical signals may be used to generate image data. Consistent withthe present disclosure, the image data may include pixel data streams,digital images, digital video streams, data derived from capturedimages, and data that may be used to construct a 3D image. The imagedata acquired by the image sensor may be transmitted by wired orwireless transmission to a remote server.

Consistent with the present disclosure, the image sensor may be part ofa camera included in a mobile communications device. The term “mobilecommunications device” refers to any portable device with imagecapturing capabilities that can communicate with a remote server over awireless network. Examples of mobile communications devices include,smartphones, tablets, smartwatches, smart glasses, wearable sensors andother wearable devices, wireless communication chipsets, user equipment(UE), personal digital assistants, and any other portable pieces ofcommunications equipment. It is noted that the terms “handheld mobilecommunications device,” “handheld mobile device,” “mobile communicationsdevice,” and “mobile device” may be used interchangeably herein and mayrefer to any of the variety of devices listed above.

Embodiments of the present disclosure further include analyzing imagesto identify a colorized surface in proximity to a medical analysisregion. As used herein, the term “colorized surface” may broadly referto any surface having planar or nonplanar properties. The colorizedsurface may cover or encapsulate at least a portion of a 2D object (suchas a sheet of paper) or at least a portion of a 3D object (such as a boxor a body part). The colorized surface may include a plurality ofreference elements for enabling light and color calibration. In someembodiments, the colorized surface may be printed on a sticker or aplaster (e.g., adhesive bandage), for example, the colorized surfaceillustrated in FIG. 4A. In other embodiments, the colorized surface maybe printed or otherwise presented on a board, cardstock, plastic or anyother medium adapted to serve as a reference. The colorized surface maybe incorporated into the packaging of a test kit, for example. Onenon-limiting example of a colorized surface is illustrated in FIG. 4B.The image correction enabled by the colorized surface may be used toenable a color correction of an image of an object depicted in themedical analysis region. As used herein, the term “medical analysisregion” may be an area on or near the surface distinct from thecolorized portion of the surface used for color correction where anobject for examination may be placed. The medical analysis region may beof uniform color or varied color so long as other portions of thecolorized surface may be used as references for color correction. In apreferred embodiment, the colorized surface may include an un-colorizedor uniformly colorized region demarcated for object placement. Such adistinct region may be larger than the object to be received thereon. Inother embodiments, the medical analysis region may not be demarcated,permitting the user to independently select a location of objectplacement, so long as enough of the colorized surface remains unblockedfor reference purposes during image analysis.

In some embodiments, the examined object is a skin or other tissue oranatomical feature, and the medical analysis region may include any partof the patient's body depicted in the image. In another embodiment, theexamined object may be a dipstick, and the color of the medical analysisregion may be significantly darker or lighter than a majority of thecolorized surface. For example, the medical analysis region may be atleast 50% darker than the colorized surface. It is noted that the terms“medical analysis region,” “dipstick placement region,” and “testregion,” may be used interchangeably herein to refer to the same area.

Consistent with the present disclosure, the colorized surface may enableprocessing of the image to determine the colors of the examined object,irrespective of local illumination conditions. The term “irrespective oflocal illumination conditions” refers to the output of an image analysisprocess in which the suggested system rectifies the colors of theexamined object to remove at least some effects of local illumination.Effects of local illumination conditions to be removed, may include oneor more of reflections, shades, light temperature (e.g., soft white,cool white, daylight), and any other condition that may impact thedetection of object color. Additionally, the colorized surface may alsoenable processing of the image to determine the colors of the examinedobject, irrespective of specific image capturing effects associated withthe image capturing device. Examples of the different effects associatedwith the image capturing process that may be removed are describedbelow.

In some embodiments, an image correction factor may be generated basedon the determined local illumination conditions and/or image capturingparameters. The image correction factor may be used to remove one ormore local illumination variations and to determine illuminationinvariant colors of the examined object. The image correction factor maybe used to remove image capturing process effects to determine capturingprocess invariant colors of the examined object. In one example, theinvariant colors may be used to determine an extent of a chemicalreaction on a reagent pad. In another example, the invariant colors maybe used to determine a skin condition, such as a condition of a wound.In yet another example, the invariant colors may be used to determine acondition of a tissue, such as skin, oral mucosa, nasal mucosa, and soforth. In an additional example, the invariant colors may be used todetermine properties of biological material, such as a stool sample, aurine sample, a phlegm sample, a blood sample, a wax sample, and soforth.

The term “confidence level” refers to any indication, numeric orotherwise, of a level (e.g., within a predetermined range) indicative ofan amount of confidence the system has that the determined colors of theexamined object are the colors of the examined object irrespective oflocal illumination conditions and/or image capturing settings effects.For example, the confidence level may have a value between 1 and 10.Alternatively, the confidence level may be expressed as a percentage orany other numerical or non-numerical indication. In some cases, thesystem may compare the confidence level to a threshold. The term“threshold” as used herein denotes a reference value, a level, a point,or a range of values. In operation, when a confidence level of ameasurement exceeds a threshold (or below it depending on a particularuse case), the system may follow a first course of action and, when theconfidence level is below it (or above it depending on a particular usecase), the system may follow a second course of action. The value of thethreshold may be predetermined for each type of examined object or maybe dynamically selected based on different considerations.

Reference is now made to FIG. 1A, which shows an example of a system 100that uses image analysis to complete a medical examination. System 100may be computer-based and may include computer system components,desktop computers, workstations, tablets, handheld computing devices,memory devices, and/or internal network(s) connecting the components.System 100 may include or be connected to various network computingresources (e.g., servers, routers, switches, network connections,storage devices, etc.) for supporting services provided by system 100.

Consistent with the present disclosure, system 100 may enable user 110to complete a medical examination. In addition, system 100 may enable amedical practitioner 120 to participate in the medical examination usinga mobile communications device 125. The disclosure below that describesthe functionalities of mobile communications device 115 similarlydescribes the functionalities of mobile communications device 125. Inone embodiment, medical practitioner 120 may be a nurse that capturesimages of an object associated with user 110. In another embodiment,medical practitioner 120 may be a physician of user 110 who receives thetest results of the medical examination. In the example illustrated inFIG. 1A, user 110 may use mobile communications device 115 to capture animage 130 that includes a colorized surface 132 and an object to beexamined 134. Image data associated with image 130 may be transmitted toa medical analysis unit 140 for medical testing (directly or via acommunication network). Medical analysis unit 140 may include a server145 coupled to one or more physical or virtual storage devices such as adata structure 146. System 100 may also include or be connected to acommunications network 150 that facilitates communications and dataexchange between different system components and the different entitiesassociated with system 100, such as, healthcare provider 160, insurancecompany 170, and pharmacy 180.

According to embodiments of the present disclosure, medical analysisunit 140 may exchange data with a variety of communication devicesassociated with the different entities associated with system 100. Theterm “communication device” is intended to include all possible types ofdevices capable of exchanging data using communications network 150. Insome examples, the communication device may include a smartphone, atablet, a mobile station, a personal digital assistant, a desktop, alaptop, an IoT device, a dedicated terminal, a server, a cloud, and anyother device that enables data communications. In one implementation,medical analysis unit 140 may receive image data from mobilecommunications device 115, and cause mobile communications device 115 toprovide user 110 with data derived from analysis of examined object 134.In another implementation, medical analysis unit 140 may transmit datato a communications device 165 of healthcare provider 160 for updatingan electronic medical record (EMR) of user 110 stored in data structure166. In another implementation, medical analysis unit 140 may receiveinformation from a communications device 175 of insurance company 170.The received information may identify a group of individuals associatedwith a first insurance status. Thereafter, medical analysis unit 140 mayinitiate medical examinations to determine if there is a likelihood thatthe group of individuals is entitled to a second insurance statusdifferent from the first insurance status. In yet anotherimplementation, medical analysis unit 140 may transmit a medicineprescription to pharmacy 180 for treating user 110 based on the testresult derived from image data captured by mobile communications device115.

Embodiments of the present disclosure may include accessing or otherwiseutilize one or more data structures, such as a database. As used hereinthe term “data structure” may include any collection of data values andrelationships among them. The data may be stored linearly, horizontally,hierarchically, relationally, non-relationally, uni-dimensionally,multidimensionally, operationally, in an ordered manner, in an unorderedmanner, in an object-oriented manner, in a centralized manner, in adecentralized manner, in a distributed manner, in a custom manner, or inany manner enabling data access. By way of non-limiting examples, datastructures may include an array, an associative array, a linked list, abinary tree, a balanced tree, a heap, a stack, a queue, a set, a hashtable, a record, a tagged union, ER model, and a graph. For example, adata structure may include an XML datastructure, an RDBMS datastructure,an SQL data structure or NoSQL alternatives for data storage/search suchas, for example, MongoDB, Redis, Couchbase, Datastax Enterprise Graph,Elastic Search, Splunk, Solr, Cassandra, Amazon DynamoDB, Scylla, HBase,SharePoint, Sybase, Oracle and Neo4J. Data structures, where suitable,may also include document management systems. A data structure may be acomponent of the disclosed system or a remote computing component (e.g.,a cloud-based data structure). Data in the data structure may be storedin contiguous or non-contiguous memory. Moreover, a data structure, asused herein, does not require information to be co-located. It may bedistributed across multiple servers, for example, that may be owned oroperated by the same or different entities. Thus, the term “datastructure” as used herein in the singular is inclusive of plural datastructures.

Consistent with the present disclosure, server 145 may access datastructure 146 to determine, for example, specific chromatic propertiesassociated with colorized surface 132 at the time of printing of thecolorized surface 132. Data structures 146 and data structure 166 mayutilize a volatile or non-volatile, magnetic, semiconductor, tape,optical, removable, non-removable, other type of storage device ortangible or non-transitory computer-readable medium, or any medium ormechanism for storing information. Data structure 146 (and datastructure 166 mutatis mutandis) may be part of server 145 or separatefrom server 145 as shown. When data structure 146 is not part of server145, server 145 may exchange data with data structure 146 via acommunication link. Data structure 146 may include one or more memorydevices that store data and instructions used to perform one or morefeatures of the disclosed embodiments. In one embodiment, data structure146 may include a plurality of suitable data structures, ranging fromsmall data structures hosted on a workstation to large data structuresdistributed among data centers. Data structure 146 may also include anycombination of one or more data structures controlled by memorycontroller devices (e.g., server(s), etc.) or software.

Consistent with the present disclosure, communications network 150 maybe any type of network (including infrastructure) that supportscommunications, exchanges information, and/or facilitates the exchangeof information between the components of system 100. For example,communications network 150 may include or be part of the Internet, aLocal Area Network, wireless network (e.g., a Wi-Fi/302.11 network), orother suitable connections. In other embodiments, one or more componentsof system 100 may communicate directly through dedicated communicationlinks, such as, for example, a telephone network, an extranet, anintranet, the Internet, satellite communications, off-linecommunications, wireless communications, transponder communications, alocal area network (LAN), a wide area network (WAN), a virtual privatenetwork (VPN), or any other mechanism or combinations of mechanisms thatenable data transmission.

The components and arrangements of system 100 shown in FIG. 1A areintended to provide examples and are not intended to limit the disclosedembodiments, as the system components used to implement the disclosedprocesses and features may vary.

FIG. 1B is a flowchart of an example process for completing a medicalexamination according to embodiments of the present disclosure. In someembodiments, the example process is executed by different components ofsystem 100. For example, healthcare provider 160, medical analysis unit140, and user 110. In one embodiment, any action performed by server 145may be performed by any combination of mobile communications device 115,mobile communications device 125, communications device 165, andcommunications device 175. FIG. 1C illustrates how the example processis implemented by healthcare provider 160, medical analysis unit 140,and user's mobile communications device 115.

Example process 190 starts when healthcare provider 160 causes a hometesting kit to be physically provided to user 110 (step 191). Consistentwith the present disclosure, causing the home testing kit to bephysically provided to user 110 may include shipping the test kit touser 110, sending an instruction to a third party to ship a test kit touser 110, physically providing user 110 with a test kit, or conveying atest to user 110 in any other way. For example, shipping instructionsmay be generated, a pick up order may be placed with a shipping company,or the testing kit may be deposited for pickup by a courier. In somecases, healthcare provider 160 may cause home testing kits to bedelivered to a group of individuals identified through information frominsurance company 170. In other cases, healthcare provider 160 may causehome testing kits to be delivered to user 110 in response to a requestfrom medical practitioner 120 or as the result of a request from user110. Alternatively, healthcare provider 160 may automatically cause hometesting kits to be delivered to user 110 based on information about user110 stored in data structure 166. In one example, a physician may havepreviously prescribed annual testing for user 110, or user 110 mighthave met some triggering time-based criteria or health-based criteriathat triggers an indication that user 110 should receive the test kit.In another example, an operator (such as a healthcare provider 160,insurance company 170, etc.) may conduct a query on data structure 166to identify users that meet the selected criteria, and may causedelivery of home testing kits to at least some of the identified users.

Process 190 may continue when user 110 sends a message confirming thereceipt of the home testing kit (step 192). In some embodiments, user110 may send the message directly to healthcare provider 160. In otherembodiments, user 110 may send the message using a dedicated applicationassociated with medical analysis unit 140, and the message may beconveyed to healthcare provider 160. The message may be text or voicebased, or may occur as a button pushed or box checked in response to aprompt on a user interface. Alternatively, the message may simply be thescanning or entry of a code. Thereafter, healthcare provider 160 maysend a verification code to user 110 (step 193). According to oneembodiment, the verification code may be sent in a text message directlyto user 110 after receiving the confirmation message, or may be providedthrough a user interface of an application accessed via a device of user110. As an alternative to an exchange of electronic messages in order toobtain the verification code, the verification code may be physicallyprovided with the home testing kit in step 191. In such an example, step192 and step 193 may be excluded from process 190.

Process 190 may continue when user 110 follows instructions associatedwith the specific medical examination, uses mobile communications device115 to capture image 130, and transmits image data together with (or ina manner that causes it to be associated with) the verification code tomedical analysis unit 140 (step 194). The image data transmitted toimage analysis unit 140 may include image 130, a cropped image withexamined object 134, a processed version of image 130 (e.g., one wherethe color of at least part of the pixels of image 130 was correctedbased on colorized surface 132), or data derived from image 130. In oneaspect of the disclosure, examined object 134 may be a skin feature.According to another aspect of the disclosure, examined object 134 mayinclude a reagent, such as a dipstick with one or more reagent pads.

Process 190 may continue when medical analysis unit 140 determines testresults associated with a state of examined object 134, possibly takinginto account local illumination conditions and/or image capturingsettings effects. In other words, medical analysis unit 140 may inspectthe image of examined object 134 after the effects of the localillumination conditions and/or of the effects of the image capturingsettings are removed. In another example, medical analysis unit 140 mayinspect the image of examined object 134 with a function that takes intoaccount local illumination conditions and/or image capturing settingseffects. When examined object 134 is a dipstick, determining its statemay include determining an extent of a chemical reaction on a least onereagent pad of the dipstick. When examined object 134 is a skin feature,determining the object's state may include determining its condition,including, for example, comparing the object's state relative to aprevious record of the skin feature. In one example, when the skinfeature is a wound, medical analysis unit 140 may determine from theimage data a healing progress of the wound. In another example, when theskin feature is a mole, medical analysis unit 140 may determine from theimage data the likelihood that the mole changed in size or that it hasan increased risk of being cancerous. Thereafter, medical analysis unit140 may transmit the test results to healthcare provider 160 (step 195),and/or to other entities (such as user 110, medical practitioner 120,insurance company 170, pharmacy 180, and so forth).

Process 190 may continue when healthcare provider 160 initiates anaction based on the received test results. In one embodiment, initiatingan action based on the received test results may include presenting thetest results to medical practitioner 120 (e.g., the user's physician).In another embodiment, initiating an action based on the received testresults may include updating an electronic medical record (EMR) of user110. In another embodiment, initiating an action based on the receivedtest results may include generating a prescription and automatically (orsemi-automatically) forwarding it to pharmacy 180. In anotherembodiment, initiating an action based on the received test results mayinclude sending medical information to user 110 (step 196) or permittingmedical analysis unit 140 to send medical information to user 110. Themedical information transmitted to user 110 may include the testresults, an invitation to schedule an appointment, a prescription, anindication that the user may be eligible for a different insurancecoverage, or any other action that results from the test.

FIG. 1C is a message flow diagram illustrating communications exchangesbetween different entities implementing the process of FIG. 1B,consistent with some embodiments of the present disclosure. It is to beunderstood that the process may be modified consistent with embodimentsdisclosed herein.

FIG. 2 is an example block diagram of configurations of server 145 andmobile communications device 115. In one embodiment, server 145 andmobile communications device 115 directly or indirectly accesses a bus200 (or other communication mechanism) that interconnects subsystems andcomponents for transferring information within server 145 and/or mobilecommunications device 115. For example, bus 200 may interconnect aprocessing device 202, a memory interface 204, a network interface 206,a peripherals interface 208 connected to I/O system 210, and powersource 209.

Processing device 202, shown in FIG. 2, may include at least oneprocessor configured to execute computer programs, applications,methods, processes, or other software to perform embodiments describedin the present disclosure. For example, the processing device mayinclude one or more integrated circuits, microchips, microcontrollers,microprocessors, all or part of a central processing unit (CPU),graphics processing unit (GPU), digital signal processor (DSP), fieldprogrammable gate array (FPGA), or other circuits suitable for executinginstructions or performing logic operations. The processing device mayinclude at least one processor configured to perform functions of thedisclosed methods such as a microprocessor. The processing device mayinclude a single core or multiple core processors executing parallelprocesses simultaneously. In one example, the processing device may be asingle core processor configured with virtual processing technologies.The processing device may implement virtual machine technologies orother technologies to provide the ability to execute, control, run,manipulate, store, etc., multiple software processes, applications,programs, etc. In another example, the processing device may include amultiple-core processor arrangement (e.g., dual, quad core, etc.)configured to provide parallel processing functionalities to allow adevice associated with the processing device to execute multipleprocesses simultaneously. It is appreciated that other types ofprocessor arrangements could be implemented to provide the capabilitiesdisclosed herein.

In some embodiments, processing device 202 may use memory interface 204to access data and a software product stored on a memory device or anon-transitory computer-readable medium. For example, server 145 may usememory interface 204 to access data structure 146. As used herein, anon-transitory computer-readable storage medium refers to any type ofphysical memory on which information or data readable by at least oneprocessor can be stored. Examples include random access memory (RAM),read-only memory (ROM), volatile memory, nonvolatile memory, harddrives, CD ROMs, DVDs, flash drives, disks, any other optical datastorage medium, any physical medium with patterns of holes, a RAM, aPROM, and EPROM, a FLASH-EPROM or any other flash memory, NVRAM, acache, a register, any other memory chip or cartridge, and networkedversions of the same. The terms “memory” and “computer-readable storagemedium” may refer to multiple structures, such as a plurality ofmemories or computer-readable storage mediums located within mobilecommunications device 115, server 145, or at a remote location.Additionally, one or more computer-readable storage mediums can beutilized in implementing a computer-implemented method. The term“computer-readable storage medium” should be understood to includetangible items and exclude carrier waves and transient signals.

Both mobile communications device 115 and server 145 may include networkinterface 206 coupled to bus 200. Network interface 206 may providetwo-way data communications to a network, such as network 150. In FIG.2, the wireless communication between mobile communications device 115and server 145 is represented by a dashed arrow. In one embodiment,network interface 206 may include an integrated services digital network(ISDN) card, cellular modem, satellite modem, or a modem to provide adata communication connection over the Internet. As another example,network interface 206 may include a wireless local area network (WLAN)card. In another embodiment, network interface 206 may include anEthernet port connected to radio frequency receivers and transmittersand/or optical (e.g., infrared) receivers and transmitters. The specificdesign and implementation of network interface 206 may depend on thecommunications network(s) over which mobile communications device 115and server 145 are intended to operate. For example, in someembodiments, mobile communications device 115 may include networkinterface 206 designed to operate over a GSM network, a GPRS network, anEDGE network, a Wi-Fi or WiMax network, and a Bluetooth® network. In anysuch implementation, network interface 206 may be configured to send andreceive electrical, electromagnetic or optical signals that carrydigital data streams representing various types of information.

Both mobile communications device 115 and server 145 may also includeperipherals interface 208 coupled to bus 200. Peripherals interface 208may be connected to sensors, devices, and subsystems to facilitatemultiple functionalities. In one embodiment, peripherals interface 208may be connected to I/O system 210 configured to receive signals orinput from devices and to provide signals or output to one or moredevices that allow data to be received and/or transmitted by mobilecommunications device 115 and server 145. In one example, I/O system 210may include a touch screen controller 212, audio controller 214, and/orother input controller(s) 216. Touch screen controller 212 may becoupled to a touch screen 218. Touch screen 218 and touch screencontroller 212 can, for example, detect contact, movement or breakthereof using any of a plurality of touch sensitivity technologies,including but not limited to capacitive, resistive, infrared, andsurface acoustic wave technologies as well as other proximity sensorarrays or other elements for determining one or more points of contactwith the touch screen 218. Touch screen 218 can also, for example, beused to implement virtual or soft buttons and/or a keyboard. While atouch screen 218 is shown in FIG. 2, I/O system 210 may include adisplay screen (e.g., CRT or LCD) in place of touch screen 218. Audiocontroller 214 may be coupled to a microphone 220 and a speaker 222 tofacilitate voice-enabled functions, such as voice recognition, voicereplication, digital recording, and telephony functions. The other inputcontroller(s) 216 may be coupled to other input/control devices 224,such as one or more buttons, rocker switches, thumbwheel, infrared port,USB port, and/or a pointer device such as a stylus.

With regard to mobile communications device 115, peripherals interface208 may also be connected to an image sensor 226, a motion sensor 228, alight sensor 230, and/or a proximity sensor 232 to facilitate imagecapturing, orientation, lighting, and proximity functions. Other sensors(not shown) can also be connected to the peripherals interface 208, suchas a temperature sensor, a biometric sensor, or other sensing devices tofacilitate related functionalities. In addition, a GPS receiver can alsobe integrated with, or connected to, mobile communications device 115,such as GPS receivers typically integrated into mobile communicationsdevices. Alternatively, GPS software may permit a mobile communicationsdevice to access AN external GPS receiver (e.g., connecting via a serialport or Bluetooth).

Consistent with the present disclosure, mobile communications device 115may use memory interface 204 to access memory device 234. Memory device234 may include high-speed random-access memory and/or non-volatilememory such as one or more magnetic disk storage devices, one or moreoptical storage devices, and/or flash memory (e.g., NAND, NOR). Memorydevice 234 may store an operating system 236, such as DARWIN, RTXC,LINUX, iOS, UNIX, OSX, WINDOWS, or an embedded operating system such asVXWorkS. The operating system 236 can include instructions for handlingbasic system services and for performing hardware-dependent tasks. Insome implementations, the operating system 236 can be a kernel (e.g.,UNIX kernel).

Memory device 234 may also store communication instructions 238 tofacilitate communicating with one or more additional devices, one ormore computers and/or one or more servers. Memory device 234 caninclude: graphical user interface instructions 240 to facilitate graphicuser interface processing; sensor processing instructions 242 tofacilitate sensor-related processing and functions; phone instructions244 to facilitate phone-related processes and functions; electronicmessaging instructions 246 to facilitate electronic-messaging relatedprocesses and functions; web browsing instructions 248 to facilitate webbrowsing-related processes and functions; media processing instructions250 to facilitate media processing-related processes and functions;GPS/navigation instructions 252 to facilitate GPS and navigation-relatedprocesses and instructions; capturing instructions 254 to facilitateprocesses and functions related to image sensor 226; and/or othersoftware instructions 258 to facilitate other processes and functions.Memory device 234 may also include application specific instructions 260to facilitate a process for guiding user 110 on the steps of the medicaltesting. For example, application specific instructions 260 may causedisplay of a message indicative of image insufficiency for medicaltesting.

Each of the above identified instructions and applications maycorrespond to a set of instructions for performing one or more functionsdescribed above. These instructions need not be implemented as separatesoftware programs, procedures, or modules. Memory device 234 may includeadditional instructions or fewer instructions. Furthermore, variousfunctions of mobile communications device 115 may be implemented inhardware and/or in software, including in one or more signal processingand/or application specific integrated circuits. For example, mobilecommunications device 115 may execute an image processing algorithm toidentify objects in a received image. In addition, the components andarrangements shown in FIG. 2 are not intended to limit the disclosedembodiments. As will be appreciated by a person skilled in the arthaving the benefit of this disclosure, numerous variations and/ormodifications may be made to the depicted configuration of server 145.For example, not all components may be essential for the operation ofserver 145 in all cases. Any component may be located in any appropriatepart of server 145, and the components may be rearranged into a varietyof configurations while providing the functionality of the disclosedembodiments. For example, some servers may not include all of theelements in I/O system 210.

A convolution may include a convolution of any dimension. Aone-dimensional convolution is a function that transforms an originalsequence of numbers to a transformed sequence of numbers. Theone-dimensional convolution may be defined by a sequence of scalars.Each particular value in the transformed sequence of numbers may bedetermined by calculating a linear combination of values in asubsequence of the original sequence of numbers corresponding to theparticular value. A result value of a calculated convolution may includeany value in the transformed sequence of numbers. Likewise, ann-dimensional convolution is a function that transforms an originaln-dimensional array to a transformed array. The n-dimensionalconvolution may be defined by an n-dimensional array of scalars (knownas the kernel of the n-dimensional convolution). Each particular valuein the transformed array may be determined by calculating a linearcombination of values in an n-dimensional region of the original arraycorresponding to the particular value. A result value of a calculatedconvolution may include any value in the transformed array.

In some embodiments, machine learning algorithms (also referred to asmachine learning models in the present disclosure) may be trained usingtraining examples, for example in the cases described below. Somenon-limiting examples of such machine learning algorithms may includeclassification algorithms, data regressions algorithms, imagesegmentation algorithms, visual detection algorithms (such as objectdetectors, face detectors, person detectors, motion detectors, edgedetectors, etc.), visual recognition algorithms (such as facerecognition, person recognition, object recognition, etc.), speechrecognition algorithms, mathematical embedding algorithms, naturallanguage processing algorithms, support vector machines, random forests,nearest neighbors algorithms, deep learning algorithms, artificialneural network algorithms, convolutional neural network algorithms,recurrent neural network algorithms, linear machine learning models,non-linear machine learning models, ensemble algorithms, and so forth.For example, a trained machine learning algorithm may comprise aninference model, such as a predictive model, a classification model, adata regression model, a clustering model, a segmentation model, anartificial neural network (such as a deep neural network, aconvolutional neural network, a recurrent neural network, etc.), arandom forest, a support vector machine, and so forth. In some examples,the training examples may include example inputs together with thedesired outputs corresponding to the example inputs. Further, in someexamples, training machine learning algorithms using the trainingexamples may generate a trained machine learning algorithm, and thetrained machine learning algorithm may be used to estimate outputs forinputs not included in the training examples. In some examples,engineers, scientists, processes and machines that train machinelearning algorithms may further use validation examples and/or testexamples. For example, validation examples and/or test examples mayinclude example inputs together with the desired outputs correspondingto the example inputs, a trained machine learning algorithm and/or anintermediately trained machine learning algorithm may be used toestimate outputs for the example inputs of the validation examplesand/or test examples, the estimated outputs may be compared to thecorresponding desired outputs, and the trained machine learningalgorithm and/or the intermediately trained machine learning algorithmmay be evaluated based on a result of the comparison. In some examples,a machine learning algorithm may have parameters and hyper parameters,where the hyper parameters may be set manually by a person orautomatically by an process external to the machine learning algorithm(such as a hyper parameter search algorithm), and the parameters of themachine learning algorithm may be set by the machine learning algorithmbased on the training examples. In some implementations, thehyper-parameters may be set based on the training examples and thevalidation examples, and the parameters may be set based on the trainingexamples and the selected hyper-parameters. For example, given thehyper-parameters, the parameters may be conditionally independent of thevalidation examples.

In some embodiments, trained machine learning algorithms (also referredto as machine learning models and trained machine learning models in thepresent disclosure) may be used to analyze inputs and generate outputs,for example in the cases described below. In some examples, a trainedmachine learning algorithm may be used as an inference model that whenprovided with an input generates an inferred output. For example, atrained machine learning algorithm may include a classificationalgorithm, the input may include a sample, and the inferred output mayinclude a classification of the sample (such as an inferred label, aninferred tag, and so forth). In another example, a trained machinelearning algorithm may include a regression model, the input may includea sample, and the inferred output may include an inferred valuecorresponding to the sample. In yet another example, a trained machinelearning algorithm may include a clustering model, the input may includea sample, and the inferred output may include an assignment of thesample to at least one cluster. In an additional example, a trainedmachine learning algorithm may include a classification algorithm, theinput may include an image, and the inferred output may include aclassification of an item depicted in the image. In yet another example,a trained machine learning algorithm may include a regression model, theinput may include an image, and the inferred output may include aninferred value corresponding to an item depicted in the image (such asan estimated property of the item, such as size, volume, age of a persondepicted in the image, cost of a product depicted in the image, and soforth). In an additional example, a trained machine learning algorithmmay include an image segmentation model, the input may include an image,and the inferred output may include a segmentation of the image. In yetanother example, a trained machine learning algorithm may include anobject detector, the input may include an image, and the inferred outputmay include one or more detected objects in the image and/or one or morelocations of objects within the image. In some examples, the trainedmachine learning algorithm may include one or more formulas and/or oneor more functions and/or one or more rules and/or one or moreprocedures, the input may be used as input to the formulas and/orfunctions and/or rules and/or procedures, and the inferred output may bebased on the outputs of the formulas and/or functions and/or rulesand/or procedures (for example, selecting one of the outputs of theformulas and/or functions and/or rules and/or procedures, using astatistical measure of the outputs of the formulas and/or functionsand/or rules and/or procedures, and so forth).

In some embodiments, artificial neural networks may be configured toanalyze inputs and generate corresponding outputs, for example in thecases described herein. Some non-limiting examples of such artificialneural networks may comprise shallow artificial neural networks, deepartificial neural networks, feedback artificial neural networks, feedforward artificial neural networks, autoencoder artificial neuralnetworks, probabilistic artificial neural networks, time delayartificial neural networks, convolutional artificial neural networks,recurrent artificial neural networks, long short term memory artificialneural networks, and so forth. In some examples, an artificial neuralnetwork may be configured manually. For example, a structure of theartificial neural network may be selected manually, a type of anartificial neuron of the artificial neural network may be selectedmanually, a parameter of the artificial neural network (such as aparameter of an artificial neuron of the artificial neural network) maybe selected manually, and so forth. In some examples, an artificialneural network may be configured using a machine learning algorithm. Forexample, a user may select hyper-parameters for the an artificial neuralnetwork and/or the machine learning algorithm, and the machine learningalgorithm may use the hyper-parameters and training examples todetermine the parameters of the artificial neural network, for exampleusing back propagation, using gradient descent, using stochasticgradient descent, using mini-batch gradient descent, and so forth. Insome examples, an artificial neural network may be created from two ormore other artificial neural networks by combining the two or more otherartificial neural networks into a single artificial neural network.

Some non-limiting examples of image data may include images, grayscaleimages, color images, 2D images, 3D images, videos, 2D videos, 3Dvideos, frames, footages, data derived from other image data, and soforth. In some embodiments, analyzing image data (for example in thecases described herein) may comprise analyzing the image data to obtaina preprocessed image data, and subsequently analyzing the image dataand/or the preprocessed image data to obtain the desired outcome. One ofordinary skill in the art will recognize that the followings areexamples, and that the image data may be preprocessed using other kindsof preprocessing methods. In some examples, the image data may bepreprocessed by transforming the image data using a transformationfunction to obtain a transformed image data, and the preprocessed imagedata may comprise the transformed image data. For example, thetransformed image data may comprise one or more convolutions of theimage data. For example, the transformation function may comprise one ormore image filters, such as low-pass filters, high-pass filters,band-pass filters, all-pass filters, and so forth. In some examples, thetransformation function may comprise a nonlinear function. In someexamples, the image data may be preprocessed by smoothing at least partsof the image data, for example using Gaussian convolution, using amedian filter, and so forth. In some examples, the image data may bepreprocessed to obtain a different representation of the image data. Forexample, the preprocessed image data may comprise: a representation ofat least part of the image data in a frequency domain; a DiscreteFourier Transform of at least part of the image data; a Discrete WaveletTransform of at least part of the image data; a time/frequencyrepresentation of at least part of the image data; a representation ofat least part of the image data in a lower dimension; a lossyrepresentation of at least part of the image data; a losslessrepresentation of at least part of the image data; a time ordered seriesof any of the above; any combination of the above; and so forth. In someexamples, the image data may be preprocessed to extract edges, and thepreprocessed image data may comprise information based on and/or relatedto the extracted edges. In some examples, the image data may bepreprocessed to extract image features from the image data. Somenon-limiting examples of such image features may comprise informationbased on and/or related to: edges; corners; blobs; ridges; ScaleInvariant Feature Transform (SIFT) features; temporal features; and soforth. In some examples, analyzing the image data may includecalculating at least one convolution of at least a portion of the imagedata, and using the calculated at least one convolution to calculate atleast one resulting value and/or to make determinations,identifications, recognitions, classifications, and so forth.

In some embodiments, analyzing image data (for example in the casesdescribed herein) may comprise analyzing the image data and/or thepreprocessed image data using one or more rules, functions, procedures,artificial neural networks, object detection algorithms, face detectionalgorithms, visual event detection algorithms, action detectionalgorithms, motion detection algorithms, background subtractionalgorithms, inference models, and so forth. Some non-limiting examplesof such inference models may include: an inference model preprogrammedmanually; a classification model; a regression model; a result oftraining algorithms, such as machine learning algorithms and/or deeplearning algorithms, on training examples, where the training examplesmay include examples of data instances, and in some cases, a datainstance may be labeled with a corresponding desired label and/orresult; and so forth. In some embodiments, analyzing image data (forexample in the cases described herein) may comprise analyzing pixels,voxels, point cloud, range data, etc. included in the image data.

As mentioned above, one of the challenges of turning a smartphone into aregulatory-approved clinical device is the lack of uniformity of imagecapture capabilities of smartphones. FIG. 3 illustrates twocommunication devices 115 capturing the same object. When a first mobilecommunications device 115A captures examined object 134 in proximity tocolorized surface 132, a first image 130A is acquired. When a secondmobile communications device 115B captures examined object 134 inproximity to colorized surface 132, a second image 130B is acquired.First image 130A may be different from second image 130B due todifferences between the incorporated image sensors, differences inlighting conditions from different perspectives, and/or differences inimage sensor settings. For example, first image 130A may be differentfrom second image 130B because first mobile communications device 115Ahas different white balance settings and different color correctionprofiles than second mobile communications device 115B. The whitebalance settings may be associated with how communications devices 115A,115B determine the white point for the image and if any tint should beapplied to the other colors. The color correction profile may beassociated with how communication devices 115A, 115B process colorsaturation, black levels, highlights, and the contrast of colors in theimage. In another example, first image 130A may be different from secondimage 130B because first mobile communications device 115A has differenthardware (such as image sensor resolution, dimensions, filters, colorfilters, lenses, crop factor, sensitivity, and so forth) thancommunications device 115B. In yet another example, first image 130A maybe different from second image 130B because first mobile communicationsdevice 115A has different camera configurations (such as exposure time,shutter speed, aperture, ISO, and so forth) than communications device115B.

Consistent with the present disclosure, each of image 130A and image130B may undergo an image correction process 300. Image correctionprocess 300 may include, for example, one or more steps to remove (or tocompensate for) local illumination effects and image capturing settingseffects. The local illumination effects may result from the type oflight source used to light the object, the distance of the object fromthe light source, a viewing angle of the object, position of the object,ambient light conditions, flash usage, exposure time, and so forth. Theimage capturing settings effects result from the type of image sensor226 used to capture the object, image resolution, frame rate, gain, ISOspeed, stereo base, lens, focus, zoom, color correction profile, and soforth. In some embodiments of the disclosure, correcting captured image130 may include reversing any of the tone mapping, color enhancement,white balance, and contrast enhancing of image 130. In addition,correcting image 130 may include simulating standard illuminationconditions and reduce shading and specularity effects.

Image correction process 300 is enabled through the use of colorizedsurface 132. Specifically, the qualities of one or more color swaths oncolorized surface 132 may be known in advance. To the extent differencesare detected between the actual colors of colorized surface 132 and animage such as image 130A or image 130B, the system may calculate acorrection factor necessary to rectify any such differences, and thenapply that correction factor to the captured image of object 134.

Image correction process 300 may correct each of image 130A and image130B differently. For example, image correction process 300 may includeincreasing the red color in image 130A and adding brightness to image130B. After images 130A and 130B separately undergo image correctionprocess 300, system 100 may independently determine test results 302from each of image 130A and image 130B. In accordance with the presentdisclosure, even though image 130A may be different from image 130B,test results 302 will be the same because both images captured the sameknown colorized surface 132 whose colorization is known in advance, andwhich may be used as a basis for generating different correction factorsfor the varying differences. In some embodiments, system 100 may correctone or more of captured images 130A, 130B using metadata associated withthe mobile communications device that captured one or more of capturedimages 130A, 130B. In other embodiments, system 100 may correct one ormore of captured images 130A, 130B without using any information aboutthe mobile communications device that captured one or more of capturedimages 130A, 130B.

FIG. 4A depicts one embodiment where the examined object is a skinfeature 400. Consistent with this aspect, system 100 is configured tomeasure the distribution of colors of skin feature 400 by comparing themto the colors on colorized surface 132. The colors on colorized surface132 may be selected to include at least some of the expected range ofcolors of the examined object under various illumination and capturingconditions. It may also include a range of colors from which acorrection factor may be generated. As illustrated in FIG. 4A, colorizedsurface 132 may include a plurality of colored reference elements 405and may be attachable onto a skin area next to skin feature 400. Incertain embodiments, colorized surface 132 may have different formsadapted to a medical condition of user 110 or an expected form andcharacteristics of skin feature 400. In addition, colorized surface 132may have different forms adapted to the expected capturing parameters(e.g., to capturing geometry). For example, colorized surface 132 may beround, elongated, curved, have one or more openings therein toaccommodate skin feature 400, etc.

Consistent with the present disclosure, colorized surface 132 may haveone or more colored reference elements 405 used for calibratingillumination and capturing conditions rather than or in addition torelating to colored reference elements 405 associated with the expectedcolors in skin feature 400. When skin feature 400 and colorized surface132 are captured in a single image, system 100 may determine the truecolors of captured skin feature 400 by correcting image 130. In someembodiments, colorized surface 132 may also include one or morepositioning marks 410 that may be used for image processing purposesand/or for positioning colorized surface 132 accurately with respect toskin feature 400. Moreover, positioning marks 410 may provide areference of a known dimension that may be used to estimate a size,orientation, and/or a form of skin feature 400. In certain embodiments,dimensional marks 410 may be used (e.g., by image analysis unit 140) tocorrect captured image 130 with respect to dimensions and forms and toderive an analysis of size and/or form of skin feature 400 and possiblyof other image features. For example, image analysis unit 140 maycompute the color constancy to determine whether two pixels have thesame color in the real world regardless of illumination conditionsand/or camera parameters.

In some embodiments, system 100 may provide two dimensional measurementsof different sections of skin feature 400 associated with a same color,such as size and shape characteristics (symmetry, boundary length etc.).In additional embodiments, system 100 may track skin feature parametersover time by repeatedly capturing the same skin feature over time. Inthis regard, the dimensional mark may assist in determining variationsover time. In one example, skin feature 400 may include scar tissue or arash that may be monitored daily to track healing progress. In anotherexample, skin feature 400 may be captured weekly or even monthly formonitoring potentially cancerous features or developments. Whencollecting such data over a period of time, an additional step may beadded for verifying that the correction of image 130 is consistentacross the time period in which the data was collected. Correcting image130 may further include taking into account illumination conditions andcapturing parameters associated with previously captured images.Additional details on the first aspect of the disclosure are describedin Applicant's U.S. Pat. No. 10,362,984, which is incorporated herein byreference in its entirety.

FIG. 4B provides an example of a colorized surface for use with adipstick 450 having at least one reagent pad 455. In use, system 100 maybe configured to measure the extent of a chemical reaction on at leastone reagent pad 455 by comparing a color of a reagent pad with thecalibration elements 470 on colorized surface 132. The calibrationelements on colorized surface 132 may be selected to represent at leastsome of the expected range of colors of the examined object undervarious illumination and capturing conditions. As illustrated in FIG.4B, colorized surface 132 may include a dipstick placement region 460and a plurality of calibration elements 470 located on opposing sides ofdipstick placement region 460.

In some embodiments, colorized surface 132 may include a plurality ofgrey elements 465A and 465B that may be used for determining localillumination conditions. Colorized surface 132 may also include aplurality of colored reference elements that may have been selected tocorrespond to expected colors of dipstick 450 under various differentpossible illumination conditions, capturing devices, and imageprocessing abilities of mobile communications devices 115. FIG. 4B showsa non-limiting example of colorized surface 132 exhibiting a grid ofcube-like grey elements 465 having three sides, each having a differentshade of grey, and a plurality of hexagon-shaped colored referenceelements 470 used as reference values for image color correction. On thedepicted colorized surface 132, at least two groups of grey elementswith the same shade scheme (e.g., group of grey elements 465A and groupof grey elements 465B) and at least two groups of colored referenceelements with the same color scheme (e.g., a group of colored referenceelements 470A and a group of colored reference elements 470B) may belocated on opposing sides of dipstick placement region 460.

According to some embodiments, colorized surface 132 may be providedwith geometrical elements that differ from geometrical shapes containedon the dipstick to enable differentiation between colored reagents onthe dipstick and elements on the colorized surface. Some elements oncolorized surface 132 may exhibit various shades of gray for improvedgamma correction. Moreover, colorized surface 132 may be provided withcalibration elements 470 surrounded by borders for minimizing oversmoothing of certain colors by some camera models. Additionally,colorized surface 132 may be provided with high contrast elements 475for enabling fast binary large object (BLOB) based color boardrectification on mobile communications device 115.

Aspects of this disclosure may relate to systems, methods, devices, andcomputer readable media storing instructions for generating crosssection views of a wound. As used herein, a cross section of a wound mayrefer to a depiction of a surface that is or may be exposed by a planecutting through the wound transversely, including, for example, at aright angle or substantially at a right angle of an axis. In oneexample, the depicted surface may be substantially perpendicular to asurface of a skin of a patient and/or to a surface of the wound. Inanother example, the depicted surface may be at a non-zero angle to thesurface of a skin of the patient and/or to the surface of the wound (forexample, at an angle larger than 1 degree, larger than 5 degrees, largerthan 15 degrees, larger than 30 degrees, larger than 45 degrees, largerthan 75 degrees, and so forth). A wound may include any injury to thehuman body. For example, wounds may be open wounds resulting frompenetration (e.g., puncture wounds, surgical wounds and incisions,thermal, chemical, or electric burns, bites and stings, gunshot wounds,etc.) and/or blunt trauma (e.g., abrasions, lacerations, skin tears), orthey may include closed wounds (e.g., contusions, blisters, seromas,hematomas, crush injuries, etc.). Some non-limiting examples of a woundmay include a chronic wound, acute wounds, ulcer (such as venous ulcer,arterial ulcer, diabetic ulcer, pressure ulcer, etc.), infectious wound,ischemic wound, surgical wound, radiation poisoning wound, and so forth.By way of example, server 145 of FIGS. 1A and 2 may be configured togenerate one or more cross section views of a wound 500 shown in FIG.5A.

Embodiments consistent with the present disclosure may include receiving3D information of a wound based on information captured using an imagesensor associated with an image plane substantially parallel to thewound. Some non-limiting examples of such image sensor may include colorimage sensor, grayscale image sensor, stereoscopic image sensor, activestereo image sensor, time-of-flight image sensor, structure from motionsensor, and so forth. In some embodiments, 3D information of a wound mayrefer to any data which may describe a three-dimensional shape or formof a wound. Some non-limiting examples of such 3D information mayinclude stereoscopic images, depth images, range images, arrays ofvoxels, geometric models (such as a manifold modeling the outer surfaceof the wound), polygon meshes, point clouds, and so forth. For instance,3D information of the wound may include image data such as pixel datastreams, digital images, digital video streams, and data derived from ananalysis of images captured using the image sensor, and/or written dataprovided in a numerical or textual manner such as a length or depth of awound. The 3D information of the wound may be received via a wired orwireless transmission from an external device, such as a mobilecommunications device, as described in greater detail herein. In someembodiments, the 3D information may be extracted or otherwise determinedbased on information captured using an image sensor. An image sensor maybe part of a camera included in a mobile communications device, asdescribed in greater detail herein. In some embodiments, the capturedinformation may be associated with an image plane substantially parallelto the surface of the wound. That is, the information may be captured bya device which is in a plane parallel to the surface of the wound suchthat the resulting captured image is in an image plane parallel to thesurface of the wound, i.e., the image plane and the wound plane at thesurface of the wound are planes in space that will never intersect.Alternatively, the image plane may be only substantially parallel to thewound, in which case, the planes may intersect at a point far away fromthe wound. In some embodiments, the angle created by the intersection ofthe substantially parallel image and wound planes may be less than 1°,less than 2°, less than 5°, less than 10°, less than 20°, or less than30°. By way of example, server 145 of FIG. 2 may receive 3D informationof wound 500 of FIG. 5A based on one or more images or a video capturedby image sensor 226 of mobile communications device 115 viacommunications network 150 of FIG. 1A. Mobile communications device 115may be in an image plane 502 parallel to a wound plane 504 such that animage or video captured by image sensor 226 may be parallel to woundplane 504. Alternatively, as shown in FIG. 5B, mobile communicationsdevice 115 may be at an angle 506 to wound plane 504 such that theresulting image or video captured by image sensor 226 is substantiallyparallel to wound plane 504.

In some embodiments, the 3D information of the wound may include atleast one of a range image, a stereoscopic image, a volumetric image, ora point cloud. A range image may refer to a 2D image showing thedistance to points in a scene from a specific point, wherein each pixelof the image may express the distance between a known reference frameand a visible point in the scene. A stereoscopic image may refer to twonearly identical images which may be paired to produce the illusion of asingle three-dimensional image. A volumetric image may refer to a 3Darray of voxels, each voxel representing a 3D area of the scene. A pointcloud may refer to a set of data points in space which may represent asample of point from a three-dimensional shape or object. In someembodiments, the 3D information of the wound may include at least one ofa plurality of 2D images of the wound captured from different angles, astereoscopic image of the wound, an image captured using an activestereo camera, or an image captured using a time-of-flight camera. Anactive stereo camera may refer to a device which may employ a light suchas a laser or a structured light to simplify the process of findingpixels in the multiscopic views that correspond to the same 3D point inthe scene. A time-of-flight camera may refer to a range imaging camerasystem which may employ time-of-flight techniques to resolve distancesbetween the camera and the subject for each point of the image. By wayof example, mobile communications device 115 of FIG. 5A may take orrender from one or more images or videos one or more of a range image, astereoscopic image, a volumetric image, or a point cloud. Additionallyor alternatively, a user operating mobile communications device 115 maytake a plurality of images or videos at a plurality of angles withrespect to the wound. For instance, wound 600 of FIG. 6 may be on a 3Dsurface and may not fit perfectly on one plane, therefore, mobilecommunications device 115 may rotate around an arm 602 to take a videoor more than one picture to capture wound 600 at different angles whileremaining parallel to wound 600.

In some embodiments, receiving the 3D information of the wound mayinclude one or more of analyzing a video of the wound captured using theimage sensor while the image sensor is moving, analyzing a video of thewound depicting a motion of the wound, or analyzing at least one imagecaptured using the image sensor. In one example, analyzing a video or atleast one image to obtain the 3D information may include usage ofstructure from motion algorithms. In another example, analyzing thevideo or the at least one image to obtain the 3D information may includeanalyzing the video or the at least one image using computer stereovision algorithms. In some examples, a machine learning model may betrained using training examples to determine 3D information of woundsfrom images and/or videos of the wounds. An example of such trainingexample may include a sample one or more images of a sample wound and/ora sample video of the sample wound, together with 3D informationcorresponding to the sample wound. The trained machine learning modelmay be used to analyze at least one of the video of the wound capturedusing the image sensor while the image sensor is moving, the video ofthe wound depicting a motion of the wound, or the at least one imagecaptured using the image sensor to determine the 3D information. In someembodiments, a user may move the image sensor while recording a videosuch that the produced video captures the wound at a plurality of pointsin space at one or more image planes substantially parallel to thewound. Additionally or alternatively, the user may produce the videosuch that it captures a motion of the wound, for example while the imagesensor is static or while the image sensor is also moving. For example,in FIG. 6, a video and/or a series of images of wound 600 may becaptured by an image sensor includes in mobile communications device 115while wound 600 moves with arm 602. The image sensor included in mobilecommunications device 115 may or may not move while capturing the videoand/or the series of images. Further, a user may capture at least oneimage via the image sensor and the at least one image may be analyzedseparately or in conjunction with a video.

Embodiments consistent with the present disclosure may includegenerating a cross section view of the wound by analyzing the 3Dinformation. A cross section view of a wound may refer to a depiction,for example in a 2D image or in a curve showing the depth of the woundalong the cross section, of a surface that is or would be exposed bymaking a straight cut through the wound by a plane at a right angle orsubstantially at a right angle of an axis. The generated cross sectionview of the wound may include a plurality of parallel cross sectionviews of the wound, for example at parallel planes intersecting thewound at different points. In some examples, a machine learning model(for example, a generative model, such as a generative adversarialnetwork, a transformers based model, etc.) may be trained using trainingexamples to generate cross section views of wounds from 3D information.An example of such training example may include a sample 3D informationof a sample wound together with an indication of a desired sample crosssection of the sample wound (such as a geometric parameters of a surfaceof the desired cross section), together with the desired cross sectionview of the sample wound corresponding to the desired sample crosssection, for example in a form of a 2D image. The trained machinelearning model may be used to analyze the 3D information and generatethe cross section view. In some examples, the 3D information may includea 3D array of voxels. In one example, the analysis of the 3D informationmay include determining a pixel of the cross section view by selecting acorresponding voxel in the 3D array and determining the value of thepixel as a function (such as an identity function, a monotonic function,a non-monotonic function, etc.) of the selected voxel. In anotherexample, the analysis of the 3D information may include determining apixel of the cross section view by calculating a 3D convolution of atleast some of the voxels of the 3D array. In some examples, the 3Dinformation may include a depth image or a range image, the crosssection view may include a curve showing the depth of the wound alongthe cross section (such as a graph of depths), and the analysis of the3D information may include determining the depth of the woundrepresented at a particular location on the curve from the 3Dinformation. In one example, the determination of the depth of the woundrepresented at the particular location on the curve may includingselecting a pixel of the range image and/or depth image, and determiningthe depth of the wound represented at the particular location on thecurve to be the depth corresponding to the selected pixel (or a functionof that depth). In another example, the determination of the depth ofthe wound represented at the particular location on the curve mayincluding calculating a 2D convolution of at least part of depths in therange image and/or depth image. In some examples, the determination ofthe depth of the wound represented at the particular location on thecurve may including determining the depth from the 3D information.Generating a cross section view of the wound may refer to creating adepiction of a cross section view of the wound based on the analysis ofthe 3D information. The generated depiction may be an image, acollection of images, a 2D image, a collection of 2D images, a video, acurve showing the depth of the wound along the cross section, or anyother appropriate medium for representing a cross section view of awound. By way of example, FIG. 7 illustrates an example of a crosssection view 700 of wound 500 generated by analyzing the 3D informationgenerated by mobile communications device 115.

In some embodiments, generating the cross section of the wound mayinclude selecting a cross section of the wound from a plurality of crosssections of the wound based on the 3D information and generating thecross section view of the wound by analyzing the 3D information, thecross section view of the wound corresponding to the selected crosssection. Selecting a cross section of the wound from a plurality ofcross sections based on the 3D information may refer to determining aparticular plane intersecting the wound at a particular angle and aparticular orientation based on the received 3D information of thewound. For instance, the selected cross section of the wound maycorrespond to a deepest point of the wound, a shallowest point of thewound, a major or minor axis of the wound, an edge of the wound, an areaof the wound which appears infected, or any other area of the woundwhich may be of interest to a medical professional for further analysis.By way of example, FIGS. 8A, 8B and 8C illustrate three cross sections800, 810, and 820 of wound 500 at different points in the wound. In someexamples, generating the cross section view may include selecting across section of wound 500 corresponding to the deepest point in wound500, which may correspond to cross section 810. In other examples,generating the cross section view may include selecting a cross sectionof wound 500 corresponding to the shallowest point in wound 500, whichmay correspond to cross section 800. In yet other examples, generatingthe cross section view may include selecting a cross section of wound500 corresponding to an area of the wound which appears infected, suchas infected areas 822 of cross section 820.

In some embodiments, generating the cross section view of the wound mayinclude selecting a cross section of the wound from a plurality of crosssections of the wound based on a boundary contour of the wound andgenerating the cross section view of the wound by analyzing the 3Dinformation, the cross section view of the wound corresponding to theselected cross section. A boundary contour of the wound may refer to theperimeter of the wound. Selecting a cross section of the wound based onthe boundary contour of the wound may include selecting a cross sectionof the wound corresponding to a longest chord of a shape of the boundarycontour, a shortest chord of the shape of the boundary contour, a planeperpendicular to one of the longest chord or the shortest chord of theshape of the boundary contour, a plane tangent to the boundary contourof the wound, two or more planes at a particular distance from one ofthe longest chord or the shortest chord of the shape of the boundarycontour, or any other appropriate cross section selection which may bebased on the boundary contour of the wound. A chord may refer to astraight line segment whose endpoints both lie on the boundary contouror perimeter of the wound.

By way of example, FIG. 9 illustrates a front view of wound 500depicting a boundary contour 900 of wound 500. In some examples,generating the cross section view may include selecting a cross sectionof wound 500 corresponding to a longest chord 910. In other examples,generating the cross section view may include selecting a cross sectionof wound 500 corresponding to a shortest chord 920. In yet otherexamples, generating the cross section view may include selecting across section of wound 500 corresponding to a plane perpendicular to oneof the longest chord (e.g., chord 912) or the shortest chord (e.g.,chord 922). In other examples, generating the cross section view mayinclude selecting a cross section view of wound 500 corresponding to aplane tangent to boundary contour 900 (e.g., tangent 902).

In some embodiments, generating a cross section view of the wound mayinclude obtaining a segmentation of the wound based on a tissue type,selecting a cross section of the wound from a plurality of crosssections of the wound based on the segmentation of the wound, andgenerating the cross section view of the wound by analyzing the 3Dinformation, the cross section view of the wound corresponding to theselected cross section. Obtaining a segmentation of the wound based on atissue type may refer to receiving, generating, or otherwise acquiring adivision into separate parts or segments of the wound based on differenttissue types present in the wound. For instance, a wound may besegmented based on different areas of the wound consisting of differenttypes of tissues. Tissue types may include epithelial tissue,granulation tissue, slough tissue, eschar, necrotic tissue, scab,hematoma, tendon, ligament, bone, infected tissue, non-infected tissue,or any other type of tissue which may be found in a wound. In someembodiments, a cross section of the wound may be selected based on thesegmentation of the wound, for example, to generate a cross section viewof a wound for one or more particular tissue types, to exclude aparticular tissue type from the cross section view, or to ensure one ormore particular tissue types are present in the cross section view.

By way of example, FIG. 10 illustrates a front view of wound 500depicting a segmentation of wound 500 based on tissue type. Forinstance, wound 500 may be segmented into granulation tissue 710, sloughtissue 720, and necrosis tissue 730. In some examples, generating thecross section view may include selecting a cross section view of wound500 which includes granulation tissue, slough tissue, and necrosistissue, such as cross section 1000. In other examples, generating thecross section view may include selecting a cross section view of wound500 which includes granulation tissue and slough tissue, but notnecrosis tissue, such as cross section 1010. In yet other examples,generating the cross section view may include selecting a cross sectionview of wound 500 which includes slough tissue and necrosis tissue, butnot granulation tissue, such as cross section 1020. In other examples,generating the cross section view may include selecting a cross sectionview of wound which includes a desired group of tissues, which do notinclude tissues of a selected group of tissues, which includes a desiredcombination of tissues (such as a desired ratio of tissues), or anycombination of the above.

In some embodiments, the generated cross section view of the wound mayinclude one or more of tissue information for at least a portion of thewound, a visual indication of a wound depth, an estimated pre-wound skincontour, and/or an estimated post-wound skin contour. Tissue informationfor at least a portion of the wound may include a description of whichareas of the wound represented in the cross section view correspond towhich tissue types and any other information which may be relevant, forexample, how large each portion of the wound is or data about eachparticular tissue type. A visual indication of a wound depth mayinclude, for example, dimensions, scales, or coloration. Estimatedpre-wound and post-wound skin contours may refer to generatedestimations of the skin in the area of the wound before the woundexisted on the body and after the wound has healed. In some embodiments,the estimated pre-wound and post-wound skin contours may be determinedby analyzing the 3D information. By way of example, FIG. 7 illustratescross section view 700 of wound 500, which may include tissueinformation for wound 500. For instance, cross section view 700 depictstissue information corresponding to granulation tissue 710, sloughtissue 720, and necrosis tissue 730. Additionally or alternatively,cross section view 700 may include a visual indication of the depth ofwound 500 at that point. For instance, numerical indication 740 andscale 750 may both provide a user with an indication of a wound depth.In some embodiments, cross section view 700 may include an estimatedpre-wound skin contour 760 and/or an estimated post-wound skin contour770. In some examples, a machine learning model (for example, agenerative model, such as a generative adversarial network, atransformers based model, etc.) may be trained using training examplesto determine estimated pre-wound and/or post wound skin contours from 3Dinformation of wounds. An example of such training example may includesample 3D information of a sample wound, together with a desiredestimation of the pre-wound and/or a post-wound skin contour for thesample wound, for example in as a function assigning a pre-wound and/orthe post-wound skin depth for each position (and/or pixel) of the samplewound, as an overlay in an image of the sample wound, and so forth. Thetrained machine learning model may be used to analyze the 3D informationand determine the estimated pre-wound and/or post-wound skin contours.In some examples, the 3D information of the wound may compared with 3Dinformation of a symmetrical body part and/or of a generic body partcorresponding to the body part associated with the wound, and theestimated pre-wound and/or post-wound skin contours may be selected tomimic the symmetrical body part and/or the generic body part. In someexamples, the 3D information may include a 3D image of the wound, thearea of the wound may be removed from the 3D image (for example using asemantic segmentation algorithm), and an inpainting algorithm mayanalyze the 3D with the wound removed to generate a 3D image of thepre-wound and/or the post-wound skin. The generated image may becompared with the 3D image of the wound to determine the pre-woundand/or the post-wound skin depth.

Embodiments consistent with the present disclosure may include providingdata configured to cause a presentation of the generated cross sectionview of the wound. The provided data may include data relating to thegenerated cross section view of the wound, including the generated crosssection view of the wound itself, and data for causing a display topresent the generated cross section view of the wound to a user, forinstance, a medical professional. The data may be provided via physicalor virtual displays such as televisions, computer monitors, head-mounteddisplays, virtual reality headsets, medical monitors, broadcastreference monitors, mobile displays, smartphone displays, video walls,or any other appropriate type of display. By way of example, data may beprovided to a device such as mobile communications devices 115, 125,and/or 165 of FIG. 1A to cause a presentation of a generated crosssection view of a wound such as cross section view 700 of wound 500 ofFIG. 3.

Some embodiments of the present disclosure may include receiving imagedata captured using the image sensor and calculating a convolution of afirst part of the image data to derive a first result value of theconvolution of the first part of the image data.

In some embodiments, a depth of the wound at a first position may bedetermined based on the first result value. The depth of the wound at afirst position may refer to a distance between the surface of the skinto the lowest point in the wound at a first position corresponding tothe first part of the image data. In one example, in response to a onevalue of the first result value, a first depth of the wound at the firstposition may be determined, and in response to another value of thefirst result value, a second depth of the wound at the second positionmay be determined, the second depth may differ from the first depth. Inanother example, the determined depth of the wound at the first positionmay be a function of the first result value. Some non-limiting examplesof such function may include a linear function, a non-linear function, amonotonic function, a non-monotonic function, a polynomial function, anexponential function, a logarithmic function, and so forth. In oneexample, the function may be obtained by training a machine learningmodel using training examples to determine depth of wounds from resultvalues of convolutions. An example of such training example may includea sample result value of a convolution of at least part of a sampleimage of a sample wound, together with a label indicating the depth ofthe sample wound.

Some embodiments of the present disclosure may include calculating aconvolution of a second part of the image data to derive a second resultvalue of the convolution of the second part of the image data, thesecond part of the image data differing from the first part of the imagedata. A second part of the image data may refer to a portion of theimage data different to the first part of the image data. For instance,this may refer to a different portion of the same image, a differentimage in a plurality of images, a different portion in a video, or anyother different appropriate portion of the image data from which thedepth of the wound may be determined based on a value of a convolution.In some embodiments, a depth of the wound at a second position based onthe second result value may be determined, the second position differingfrom the first position. The depth of the wound at a second position mayrefer to the distance from the surface of the skin to the lowest pointin the wound at a second position in the wound corresponding to thesecond part of the image data and different from the first position.

Some embodiments of the present disclosure may include estimating atleast one of an original position of a skin before a formation of thewound or a future position of the skin after healing of the wound byanalyzing the 3D information, wherein the provided data may be based onat least one of the estimated original position of the skin or thefuture position of the skin. Estimating an original position of a skinbefore a formation of the wound may refer to estimating an outline ofthe skin in the affected area where the wound is currently before thewound appeared. A future position of the skin after healing of the woundmay refer to an estimate of an outline of the skin in the affected areaafter the wound is cured. In some examples, a machine learning model(for example, a generative model, such as a generative adversarialnetwork, a transformers based model, etc.) may be trained using trainingexamples to determine original positions of the skin and/or futurepositions of the skin from 3D information of wounds. An example of suchtraining example may include sample 3D information of a sample wound,together with a desired estimation of the original position of the skinand/or future position of the skin corresponding to the sample wound,for example in as a function assigning an original position of the skinand/or a future position of the skin for each position (and/or pixel) ofthe sample wound, as an overlay in an image of the sample wound (such asan image of a cross section view of the sample view), and so forth. Thetrained machine learning model may be used to analyze the 3D informationand determine the original position of the skin and/or future positionof the skin. In some examples, the 3D information of the wound maycompared with 3D information of a symmetrical body part and/or of ageneric body part corresponding to the body part associated with thewound, and the original position of the skin and/or future position ofthe skin may be selected to mimic the symmetrical body part and/or thegeneric body part. In some examples, the 3D information may include a 3Dimage of the wound, the area of the wound may be removed from the 3Dimage (for example using a semantic segmentation algorithm), and aninpainting algorithm may analyze the 3D with the wound removed togenerate a 3D image of the original position of the skin and/or futureposition of the skin. In some embodiments, the data provided configuredto cause a presentation of the generated cross section view of the woundmay be based on at least one of the estimated original position of theskin or the estimated future position of the skin. In some embodiments,at least one of estimating the original position of the skin orestimating the future position of the skin may include implementing aninpainting algorithm based on the 3D information. An inpaintingalgorithm may refer to an algorithm which may fill in missing parts ofan image to present a complete image. An inpainting algorithm may beimplemented to “fill in,” or estimate, original and future positions ofthe skin over the wound. The inpainting algorithm may be trained usingimage data from previous wounds, including images from before the wound,during different stages of the wound's healing, and after the wound hashealed. By way of example, FIG. 7 illustrates an example of an estimatedoriginal position of the skin 760 and an estimated future position ofthe skin 770.

In some embodiments, the wound may correspond to a first body part of apatient, the patient having a symmetrical body part to the first bodypart, and wherein at least one of estimating the original position ofthe skin or estimating the future position of the skin may includereceiving 3D information of the symmetrical body part and analyzing the3D information of the symmetrical body part and the 3D information ofthe wound. A body part of a patient may refer to any part of a humanbeing suffering from a wound such as an organ or an extremity. Asymmetrical body part to the first body part may refer to a body partwhich is the counterpart of the body part suffering from the wound. Forinstance, if the wound corresponds to a patient's hand, the symmetricalbody part to the first body part may be the patient's other, healthyhand. Similarly, if the wound corresponds to a patient's nose, and ifthe wound is present on one side of the nose, the symmetrical body partmay correspond to the other, healthy side of the nose. By way ofexample, estimating original position of the skin 760 and/or the futureposition of the skin 770 may include receiving 3D information via mobilecommunications device 115 of a symmetrical body part to the body partwound 500 is located. For instance, if wound 500 is located on aforearm, a user may capture one or more images and/or videos usingmobile communications device 115 of the same area of the counterparthealthy forearm and send the one or more images and/or videos to server145 for analysis.

In some embodiments, the provided data may include a depth of the woundestimated based on at least one of the estimated original position ofthe skin or the estimated future position of the skin. For instance, thedepth of the wound may be estimated by calculating the distance betweenthe lowest point in the wound and the surface of the skin in theestimated original position of the skin or the estimated future positionof the skin. The height of the surface of the skin compared to thelowest point in the wound may be different with respect to the estimatedoriginal position of the skin and the estimated future position of theskin, as scarring of the skin following healing of the wound may causethe skin to appear different. By way of example, FIG. 7 illustrates awound depth 762 corresponding to “X mm” when calculated based onestimated original position of the skin 760 and a wound depth 772corresponding to “Y mm” when calculated based on estimated futureposition of the skin 770.

In some embodiments, the generated cross section view of the wound mayinclude a visual indication of at least one of the original position ofthe skin or the future position of the skin. For example, the generatedcross section view of the wound may include an outline or some otherindication showing the original position of the skin and/or the futureposition of the skin over the wound. By way of example, FIG. 7 includesoutlines 760 and 770 depicting the original position of the skin and thefuture position of the skin over wound 500, respectively.

FIG. 11 provides a flowchart of an example process 1100 for generatingcross section views of a wound including steps 1102 through 1108. Steps1102 through 1108 may be executed by at least one processor (e.g.,processing device 202 of server 145 or mobile communications device 115of FIG. 2), consistent with some embodiments of the present disclosure.

Process 1100 may begin with step 1102. At step 1102, the at least oneprocessor may receive 3D information of a wound based on informationcaptured using an image sensor (e.g., image sensor 226 of FIG. 2)associated with an image plane substantially parallel to the wound. Theimage sensor may be associated with a mobile device, such ascommunications device 115 of FIG. 2.

Once the 3D information is received, process 1100 may proceed to step1104. At step 1104, the at least one processor may select a crosssection of the wound from a plurality of cross sections. The selectionof the cross section of the wound may be based on a plurality offactors. These factors may include, for example, the 3D information, aboundary contour of the wound, and a segmentation of the wound based ontissue type.

At step 1106, the at least one processor may generate a cross sectionview of the wound by analyzing the received 3D information, the crosssection view of the wound corresponding to the selected cross section.For instance, if the at least one processor selected a cross section ofthe wound corresponding to the deepest point of the wound based on the3D information, the at least one processor, at step 1106, may generate across section view of the wound corresponding to this selected crosssection.

Once the cross section view has been generated, process 1100 may proceedto step 1108. At step 1108, the at least one processor may provide dataconfigured to cause a presentation of the generated cross section viewof the wound (e.g., to mobile communications devices 115, 125, and/or165, or to server 145, which may include a display or may reroute thedata to an appropriate display).

Aspects of this disclosure may relate to systems, methods, devices, andcomputer readable media storing instructions for analyzing wounds usingstandard user equipment. As used herein, a wound may include any injuryto the human body. For example, wounds may be open wounds resulting frompenetration (e.g., puncture wounds, surgical wounds and incisions,thermal, chemical, or electric burns, bites and stings, gunshot wounds,etc.) and/or blunt trauma (e.g., abrasions, lacerations, skin tears), orthey may include closed wounds (e.g., contusions, blisters, seromas,hematomas, crush injuries, etc.). Some non-limiting examples of a woundmay include a chronic wound, acute wounds, ulcer (such as venous ulcer,arterial ulcer, diabetic ulcer, pressure ulcer, etc.), infectious wound,ischemic wound, surgical wound, radiation poisoning wound, and so forth.As used herein, standard user equipment may refer to any portable devicewith image capturing capabilities that can communicate with a remoteserver over a wireless network. Examples of standard user equipment mayinclude smartphones, tablets, smartwatches, smart glasses, wearablesensors and other wearable devices, wireless communication chipsets,personal digital assistants, and any other portable pieces ofcommunications equipment. It should be noted that the terms “standarduser equipment,” “user equipment,” “handheld mobile communicationsdevice,” “handheld mobile device,” “mobile communications device,” and“mobile device” may be used interchangeably herein and may refer to anyof the variety of devices listed above. By way of example, server 145 ofFIGS. 1A and 2 may be configured to analyze a wound 1200 shown in FIG.12.

Embodiments of the present disclosure may include receiving one or moreimages of a wound of a patient. In some embodiments, one or more imagesmay be received via a wired or wireless transmission from an externaldevice, such as a mobile communications device, as described in greaterdetail herein. In some other examples, receiving the one or more imagesmay include reading the one or more images from memory, capturing theone or more images using an image sensor, receiving the one or moreimages from at least one image sensor of a mobile device, and so forth.The one or more images of a wound of a patient may include picturestaken of a patient suffering from a wound, each of the picturesincluding at least a portion of the wound, and/or an area of interestfor examination of the wound, such as a healthy area surrounding thewound or a symmetrical body part to the body part suffering from thewound. By way of example, server 145 of FIG. 2 may receive one or moreimages of wound 1200 of FIG. 12 captured by image sensor 226 of mobilecommunications device 115 via communications network 150 of FIG. 1A.

In some embodiments, the one or more images may be and/or include one ormore images captured under artificial ultra-violet light, may be and/orinclude one or more images captured under artificial infrared light,and/or may be and/or include one or more images captured using aselected physical optical filter. Artificial ultra-violet light mayrefer to electromagnetic radiation in the ultra-violet range produced byan artificial source such as black lights, curing lamps, germicidallamps, mercury vapor lamps, halogen lights, high-intensity dischargelamps, fluorescent and incandescent sources, lasers, and/or any otherman-made sources of ultra-violet radiation. Artificial infrared lightmay refer to electromagnetic radiation in the infrared range produced byan artificial source such as electrical appliances, incandescent bulbs,radiant heaters, and/or any other man-made source of infrared radiation.A selected physical optical filter may refer to a device which mayselectively transmit light of different wavelengths, as discussed ingreater detail herein. By way of example, one or more images may becaptured by mobile communications device 115 under a light 1210 whichmay be, for example, an artificial ultra-violet light, an artificialinfrared light, or a standard light. The one or more images may becaptured using a selected physical optical filter 1500, as shown in FIG.15 and discussed in greater detail below.

Embodiments of the present disclosure may include analyzing the one ormore images (for example as described above) to determine a condition ofthe wound. In one example, a machine learning model may be trained usingtraining examples to determine conditions of wounds from images. Anexample of such training example may include a sample image of a samplewound, together with a label indicating the condition of the samplewound. The trained machine learning model may be used to analyze the oneor more images to determine the condition of the wound. In anotherexample, a visual classification algorithm may classify the one or moreimages to one of a plurality of alternative classes, and each class maycorrespond to a different condition of the wound. In some examples, theembodiments of the present disclosure may include analyzing the one ormore images to determine, based on at least a difference between valuesof two pixels of the one or more images, a condition of the wound. Apixel may refer to the smallest unit of a digital image or graphic whichmay be displayed and represented on a digital display device. Adifference between values of two pixels may refer to a difference in thecoloration of the two pixels, for instance, a difference in RGB colorvalues of each pixel. A condition of a wound may refer to the state ofthe wound, for instance, if the wound is infected, clean, healingadequately, fully or partially healed, showing signs of heat, redness,swelling, or any other physical state a wound may be in. In one example,a numerical value representing a different between the values of twopixels (for example, difference in intensity, difference in a particularcolor component, or another difference) may be determined by comparingthe values of the two pixels. Further, in response to a first determinednumerical value, a first condition of the wound may be determined, andin response to a second determined numerical value, a second conditionof the wound may be determined, the second condition may differ from thefirst condition. In some examples, a machine learning model may betrained using training examples to determine a condition of a wound froma difference between values of two pixels. An example of such trainingexample may include a sample difference between two pixels of a sampleimage of a sample wound, together with a label indicating the conditionof the sample wound. The trained machine learning model may be used toanalyze the difference between the values of the two pixels to determinethe condition of the wound. In one example, the condition of a wound maybe determined based on a difference between values of two pixels by, forexample, determining that a value of one pixel represents a healthyportion of a wound and comparing said value of the pixel to a value ofanother pixel of the wound, which may correspond to a non-healthyportion of the wound. By way of example, FIGS. 13A and 13B illustrate acaptured image of wound 1200 segmented into pixels. The values of theillustrated pixels may be compared, and a difference between the RGBvalues may indicate the condition of wound 1200. For instance, adifference between the values of pixels 1302 and 1304, as shown in FIG.13A, may indicate that a wound is infected. Alternatively, a differencebetween the values of pixels 1312 and 1314, as shown in FIG. 13B, mayindicate that a wound is healing adequately.

In some embodiments, an indication of a past condition of the wound at aparticular time period may be received. The particular time period maybe at least one day before the capturing of the one or more images ofthe wound. In other examples, the particular time period may be at leastone hour, at least two hours, at least one day, at least two days, atleast a week, etc., before the capturing of the one or more images ofthe wound. In some examples, receiving the indication of the pastcondition of the wound may include at least one of reading theindication from memory, receiving the indication from an externaldevice, receiving the indication from a user (for example through a userinterface), generating the indication (for example by analyzing imagesof the wound captured at the particular time period), and so forth. Insome examples, images of the wound captured at the particular timeperiod may be analyzed to determine the past condition of the wound atthe particular time period, for example as described above in relationto the analysis of the one or more images to determine the condition ofthe wound. In some embodiments, the determination of the condition ofthe wound may be based on the past condition of the wound and theanalysis of the one or more images. For example, when the condition ofthe wound determined by the analysis of the one or more images isincompatible with the past condition of the wound, further processingmay be made to correct the determination of the condition of the wound.In another example, the past condition of the wound may be used todetermine a prior probabilities for the condition of the wound, forexample using a Markov model, and the determination of the condition ofthe wound based on the analysis of the one or more images may be furtherbased on the prior probabilities. In some examples, a machine learningmodel may be trained using training examples to determine conditions ofwounds from images of the wounds and from past conditions of the wounds.An example of such training example may include a sample image of asample wound and a sample indication of a past condition of the samplewound, together with a label indicating the condition of the samplewound. The trained machine learning model may be used to analyze the oneor more images and the indication of the past condition of the wound todetermine the condition of the wound.

Embodiments of the present disclosure may include selecting an actionbased on the determined condition of the wound and initiating theselected action. In some examples, the action may include at least oneof processing the one or more images, providing instructions to a userto capture at least one additional image of the wound, and/or providingparticular information associated with the condition of the wound.Processing the one or more images may include any image analysistechniques, including the image analysis techniques discussed above.Particular information associated with the condition of the wound mayinclude any data that may provide a user with information regarding thecondition of the wound, for instance, text describing the condition ofthe wound or a visual representation of the wound with indications ofthe condition of the wound in different areas. Selecting an action basedon the determined condition of the wound may refer to choosing one ormore actions from a plurality of actions based on the condition of thewound. For instance, if a wound is determined to have a potentiallyinfected area, an action may be selected that instructs a user tocapture at least one additional image of the affected area of the wound.In one example, initiating the selected action may refer to causing adevice (for example, the device selecting the action, the deviceanalyzing the one or more images to determine the condition of thewound, an external device, and so forth) to perform the selected actionand/or providing a device with instructions relating to the action. Inanother example, initiating the selected action may include causing auser to perform the action, for example by providing the user withinstructions and/or recommendations to perform the action (for examplevisually, audibly, textually, graphically, through a user interface, andso forth).

Embodiments consistent with the present disclosure may include analyzingthe one or more images to determine at least one of a shape of thewound, a tissue composition of the wound, a depth of the wound, or apresence of an edema in a region surrounding the wound, and wherein thedetermination of the condition of the wound may be further based on thedetermined at least one of the shape of the wound, the tissuecomposition of the wound, the depth of the wound, or the presence of theedema in the region surrounding the wound. A shape of the wound mayrefer to a 2D or 3D form made by the wound on the body of a patient. Atissue composition of the wound may refer to a segmentation of the woundbased on a tissue type, such as epithelial tissue, granulation tissue,slough tissue, eschar, necrotic tissue, scab, hematoma, tendon,ligament, bone, infected tissue, non-infected tissue, or any other typeof tissue which may be found in a wound. A depth of the wound may referto the distance between a point along the bottom of a wound and one ofthe surface of the skin surrounding the wound, an estimated originalposition of the skin, or an estimated future position of the skin. Anedema may refer to swelling caused by fluid trapped in the patient'stissues. By way of example, image 1400, as depicted in FIG. 14, may beanalyzed to determine a shape of wound 1200, a tissue composition ofwound 1200 (e.g., granulation tissue 1410, slough tissue 1420, andnecrosis tissue 1430), a depth of wound 1200, and/or a presence of anedema in a region surrounding wound 1200, such as edema 1440.

In some embodiments, the one or more images may include at least a firstimage and a second image, the first image being an image captured usinga first physical optical filter and the second image being an imagecaptured using a second physical optical filter, wherein the secondphysical optical filter may differ from the first physical opticalfilter and the determination of the condition of the wound may befurther based on an analysis of the first image and the second image. Acondition of the wound may be determined based on an analysis of thefirst image and the second image due to information combined from thetwo images. For example, the first physical optical filter may enablethe capturing of visible colors in the first image, and the secondphysical optical filter may enable the capturing of infrared light inthe second image. Combining the color information from the first imageand temperature data associated with the wound from the second image mayenable a more accurate determination of the condition of the wound incomparison to the usage of any single one of the two images. In someexamples, a machine learning model may be trained using trainingexamples to determine conditions of wounds from pairs of images capturedusing different physical optical filters. An example of such trainingexample may include one sample image of a sample wound captured usingone physical optical filter and another sample image of the sample woundcaptured using another physical optical filter, together with a labelindicating the condition of the sample wound. The trained machinelearning model may be used to analyze the first image and the secondimage to determine the condition of the wound. By way of example, theone or more images may include at least a first image captured usingphysical optical filter 1500 and at least a second image captured usinga different physical optical filter.

In some embodiments, the one or more images may include at least oneimage depicting at least part of the wound and a calibration element,the calibration element including a form of a known size, a known shape,and/or a known color. In some embodiments, the determination of thecondition of the wound may be based on at least one of the known size,the known shape, or the known color of the calibration element. Acalibration element may refer to an object that may be captured with atleast a portion of a wound in an image and may be used to ascertain asize, shape, and/or color of the at least a portion of the wound. Asize, shape, and/or color of the calibration element may be known suchthat a size, shape, and/or color may be determined for the at least aportion of the wound. By way of example, image 1400 of FIG. 14 depicts acalibration element 1450, which may have a known size, shape, and color,and which may be used to determine the size, shape, and/or color of atleast a portion of wound 1200. Another example of a calibration element,as depicted in FIG. 4A, may include colorized surface 132 and/orelements 405 and/or 410 of colorized surface 132, and the image maydepict colorized surface 132 and wound 400.

In some embodiments, the one or more images may include one or moreimages of the wound captured using a mobile communications device.Embodiments consistent with the present disclosure may include causingthe mobile communications device to provide an instruction to a user ofthe mobile communications device to capture an image of the woundwithout a physical optical filter, to place a physical optical filter onthe mobile communication device, and to capture an image of the woundwith the physical optical filter. A physical optical filter may beattached to a mobile communications device to manipulate light reachinga camera included in the mobile communications device. The physicaloptical filter may be shaped to envelop at least one corner of themobile communications device while covering the camera and may includean adhesive side configured to attach the physical optical filter to themobile communications device. By way of example, the one or more imagesmay include one or more images of wound 1200 captured using mobilecommunications device 115. In some embodiments, mobile communicationsdevice 115 may provide an instruction to a user to capture an image ofwound 1200 without a physical optical filter, then place physicaloptical filter 1500, as depicted in FIG. 15, on mobile communicationsdevice 115, and capture an image of wound 1200 with physical opticalfilter 1500 placed on mobile communications device 115.

Embodiments consistent with the present disclosure may include receivingand analyzing the image of the wound captured without the physicaloptical filter and the image of the wound captured with the physicaloptical filter to determine the condition of the wound. The condition ofthe wound may be determined from the image of the wound captured withoutthe physical optical filter and the image of the wound captured with thephysical optical filter, for example by combining information combinedfrom the two images. For example, the image of the wound capturedwithout the physical optical filter may include visible colors, and theimage of the wound captured with the physical optical filter may includeinfrared data. Combining the color information from the image of thewound captured without the physical optical filter and temperature dataassociated with the wound from the infrared data may enable a moreaccurate determination of the condition of the wound in comparison tothe usage of any single one of the two images. In some examples, amachine learning model may be trained using training examples todetermine a conditions of wounds from pairs of images, each pair mayinclude an image captured with a physical optical filter and an imagecaptured without a physical optical filter. An example of such trainingexample may include one sample image of a sample wound captured withouta physical optical filter and another sample image of the sample woundcaptured using a physical optical filter, together with a labelindicating the condition of the sample wound. The trained machinelearning model may be used to analyze the image of the wound capturedwithout the physical optical filter and the image of the wound capturedwith the physical optical filter to determine the condition of thewound.

Embodiments consistent with the present disclosure may include causingthe mobile communications device to provide an instruction to the userto place a calibration element in proximity to the wound, thecalibration element including a form of a known size, a known shape, ora known color, and using at least one of the known size, the knownshape, or the known color in the analysis of the image of the woundcaptured without the physical optical filter and the image of the woundcaptured with the physical optical filter. By way of example, mobilecommunications device 115 may provide an instruction to a user to placecalibration element 1450 near wound 1200 to capture both calibration1450 and wound 1200 in the same image, such as in image 1400.

Embodiments consistent with the present disclosure may include analyzingthe one or more images to determine that an urgency level associatedwith the wound is a first level of urgency and, in response to thedetermination that the urgency level associated with the wound is thefirst level of urgency, initiating a particular action. An urgency levelassociated with the wound may refer to a degree to which a state of thewound requires immediate action or attention. For instance, for adetermination that a wound requires immediate attention to preventfurther damage, a first level of urgency may be given to the wound.Then, in response to the determination that a first level of urgency hasbeen given to the wound, a particular action may be initiated, forexample, the particular action may be configured to cause an advancementof the patient in an order of treatment. That is, a patient may be giventreatment in advance of other patients with a lower level of urgency. Onthe other hand, a wound which does not require immediate attention orless attention may be given a second, third, fourth, or any otherappropriate level of urgency. By way of example, a first level ofurgency may be determined for wound 1200 due to the presence of sloughtissue 1420, necrosis tissue 1430, and/or edema 1440, and a particularaction may be initiated, such as advancing the patient in an order oftreatment.

In some embodiments, the one or more images may include at least a firstimage and a second image, the first image being an image captured atleast one day before a capturing of the second image, wherein thedetermination that the urgency level associated with the wound is thefirst level of urgency may be based on a comparison of the wound in thefirst image with the wound in the second image, and wherein theparticular action may be initiated within one hour of the capturing ofthe second image. For instance, if a comparison of the wound in thefirst image and the wound in the second image shows that the wound isdeteriorating rapidly, a higher level of urgency may be given to thewound to initiate a particular action sooner. By way of example, apatient may capture an image of wound 1200 on a particular day, and onthe next day capture an image of wound 1200, which shows that the woundhas developed slough and necrosis tissue and an edema, and accordinglydetermine a first level of urgency should be assigned to wound 1200.

Embodiments consistent with the present disclosure may includedetermining that a confidence level associated with the determinedcondition of the wound is a first confidence level and, in response tothe determination that the confidence level associated with thedetermined condition of the wound is the first confidence level,avoiding initiating the selected action. A confidence level may refer toa degree of certainty that a determined condition of the wound isaccurate. For instance, for a given determined condition of a woundwhich is determined to accurately reflect the actual condition of thewound, a first confidence level may be associated with the determinedcondition of the wound such that a selected action may not be initiatedas no more information on the wound may be needed. Alternatively, for adetermined condition of a wound which may not accurately reflect theactual condition of the wound, a second, third, fourth, or any otherappropriate confidence level may be associated with the condition of thewound such that the selected action may be initiated, as moreinformation on the wound may be needed.

Aspects of this disclosure may relate to a kit for facilitatingcapturing of medical images. In some embodiments, the kit may include aphysical optical filter configured to be selectively attached to astandard user mobile communications device and to manipulate lightreaching a camera included in the standard user mobile communicationsdevice when attached to the standard user mobile communications device.As used herein, a kit may refer to a set of articles or equipment neededfor a specific purpose. A physical optical filter may refer to a devicewhich may selectively transmit light of different wavelengths. In someembodiments, the physical optical filter may be shaped to envelop atleast one corner of the standard user mobile communications device whilecovering the camera included in the standard user mobile communicationsdevice. In some embodiments, the physical optical filter may include anadhesive side configured to attach the physical optical filter to thestandard user mobile communications device. By way of example, a kit1600, as depicted in FIG. 16, may include physical optical filter 1500which may be affixed onto mobile communications device 115.

Consistent with disclosed embodiments, the kit may include a calibrationelement, the calibration element including a form of a known size, aknown shape, and a known color. A calibration element may refer to anobject which may be captured with at least a portion of a wound in animage and may be used to ascertain a size, shape, and/or color of the atleast a portion of the wound. A size, shape, and/or color of thecalibration element may be known such that a size, shape, and/or colormay be determined for the at least a portion of the wound. By way ofexample, kit 1600 may include calibration element 1450, which includes aknown size, shape, and color to aid in the calibration of an image.Another example of a calibration element may include colorized surface132, as depicted in FIGS. 4A and 4B.

In some embodiments, the physical optical filter may be configured toenable capturing of at least two medical images of a wound by the cameraincluded in the standard user mobile communications device, includingcapturing at least one image using the physical optical filter andcapturing at least one image without the physical optical filter. Insome embodiments, the calibration element may be configured to enablecolor calibration of the at least one image captured using the physicaloptical filter and to enable calibration of the at least one imagecaptured without the physical optical filter. For instance, a physicaloptical filter may be completely or partially removable such that acamera included in the standard user mobile communications device maycapture at least one image using the physical optical filter and atleast one image without the physical optical filter. Calibrating theimages based on the calibration element may be performed due to theknown color of the calibration element, which may be compared tosurrounding colors in the captured images to determine their true colorsand modify the captured images based on the determination.

FIG. 17 provides a flowchart of an example process 1700 for generatingcross section views of a wound including steps 1702 through 1708. Steps1702 through 1708 may be executed by at least one processor (e.g.,processing device 202 of server 145 or mobile communications device 115of FIG. 2), consistent with some embodiments of the present disclosure.

Process 1700 may begin with step 1702. At step 1702, the at least oneprocessor may receive one or more images of a wound of a patient. By wayof example, the one or more images may have been captured using imagesensor 226 of FIG. 2 and sent by mobile communications device 115 toserver 145.

Once the one or more images are received, process 1700 may proceed tostep 1704. At step 1704, the at least one processor may analyze the oneor more images to determine, based on at least a difference betweenvalues of two pixels of the one or more images, a condition of thewound.

At step 1706, the at least one processor may select an action based onthe determined condition of the wound, wherein the selected action mayinclude at least one of additional processing of the one or more images,providing instructions to a user to capture at least one additionalimage of the wound, and/or providing particular information associatedwith the condition of the wound. For instance, if the determinedcondition of the wound includes an area of the wound which is infected,the selected action may provide instructions to a user to capture atleast one additional image of the affected area of the wound. Once theaction has been selected, process 1700 may proceed to step 1708. At step1708, the at least one processor may initiate the selected action.

Aspects of this disclosure may relate to systems, methods, devices, andcomputer readable media storing instructions for generating visual timeseries views of wounds. As used herein, a wound may include any injuryto the human body. For example, wounds may be open wounds resulting frompenetration (e.g., puncture wounds, surgical wounds and incisions,thermal, chemical, or electric burns, bites and stings, gunshot wounds,etc.) and/or blunt trauma (e.g., abrasions, lacerations, skin tears), orthey may include closed wounds (e.g., contusions, blisters, seromas,hematomas, crush injuries, etc.). Some non-limiting examples of a woundmay include a chronic wound, acute wounds, ulcer (such as venous ulcer,arterial ulcer, diabetic ulcer, pressure ulcer, etc.), infectious wound,ischemic wound, surgical wound, radiation poisoning wound, and so forth.A visual time series view may refer to a series of images or othervisual representations ordered in time. By way of example, server 145 ofFIGS. 1A and 2 may be configured to generate visual time series views2300, 2310, 2320, and 2330 shown in FIG. 23 of a wound 1800, originallydepicted in FIG. 18. According to embodiments disclosed herein, one ormore images of wound 1800 may be captured by mobile communicationsdevice 115.

Embodiments of the present disclosure may include receiving at least afirst image data record and a second image data record, the first imagedata record corresponding to a first point in time and including a firstone or more images of a wound captured at the first point in time, andthe second image data record corresponding to a second point in time andincluding a second one or more images of the wound captured at thesecond point in time. As used herein, an image data record may refer toa collection of related images or information related to the images.Each image data record may be associated with a point in time. Forinstance, a user of a standard mobile communications device may captureone or more images via a camera included in the mobile communicationsdevice at a particular point in time, and the image data recordincluding the one or more images may correspond to said particular pointin time. In one example, an image data record may be and/or include avideo of a wound captured by a user using a standard mobilecommunications device, and the one or more images included in the imagedata record may be and/or include one or more frames of the video. Forexample, the video may be a video of the wound captured using thestandard mobile communications device while the standard mobilecommunications device moves. In another example, the video may be avideo of the wound captured while the wound is moving. In yet anotherexample, the video may be a video of the wound captured using an imagesensor included in the standard mobile communications device while atleast one parameter of the image sensor changes (such as zoom, focus,and so forth). In an additional example, the video may be a video of thewound captured while the illumination conditions changes. As usedherein, a standard mobile communications device may refer to anyportable device with image capturing capabilities that can communicatewith a remote server over a wireless network. Examples of standard userequipment may include smartphones, tablets, smartwatches, smart glasses,wearable sensors and other wearable devices, wireless communicationchipsets, personal digital assistants, and any other portable pieces ofcommunications equipment. It should be noted that the terms “standarduser equipment,” “user equipment,” “handheld mobile communicationsdevice,” “handheld mobile device,” “mobile communications device,” and“mobile device” may be used interchangeably herein and may refer to anyof the variety of devices listed above. By way of example, server 145 ofFIG. 2 may receive a first image data record 1900 of FIG. 19 and asecond image data record 2000 of FIG. 20, first image data record 1900corresponding to a first point in time and including four images 1910,1920, 1930, and 1940 of wound 1800 captured at the first point in time,and second image data record 2000 corresponding to a second point intime and include four images 2010, 2020, 2030, and 2040 of wound 1800captured at the second point in time. First image data record 1900 andsecond image data record 2000, depicted in FIG. 20, may be captured bythe same device (such as mobile communications device 115), or bydifferent devices.

In some embodiments, an image data record may be received via a wired orwireless transmission from an external device, such as a mobilecommunications device, as described in greater detail herein. In otherexamples, an image data record may be read from memory, may be capturedusing an image sensor, may be generated (for example, from images and/orvideos), and so forth. An image data record of a wound of a patient mayinclude pictures taken of a patient suffering from a wound, each of thepictures including at least a portion of the wound, and/or an area ofinterest for examination of the wound, such as a healthy areasurrounding the wound or a symmetrical body part to the body partsuffering from the wound. Additionally or alternatively, an image datarecord of a wound may include one or more videos of the wound and/or anarea of interest for examination of the wound.

Embodiments of the present disclosure may include obtaining an image ofthe wound from a particular point of view corresponding to the firstpoint in time by analyzing the first image data record. In someembodiments, the image of the wound from the particular point of viewcorresponding to the first point in time may be an image of the firstone or more images of the wound. In such embodiments, obtaining an imageof the wound from a particular point of view corresponding to the firstpoint in time may include selecting an image of the wound from the firstone or more images of the wound captured at the first point in time.Alternatively, the image of the wound from the particular point of viewcorresponding to the first point in time may be a simulated image of thewound based on the first image data record. Generating a simulated imageof the wound based on an image data record is described in greaterdetail below. A particular point of view may refer to a view of thewound including a particular illumination, viewing angle, orientation,image plane, distance, coloration, and/or any other property of an imagewhich may need to be controlled from one image to the next in a visualtime series view of the wound for a medical practitioner to adequatelyexamine the wound. By way of example, server 145 of FIG. 2 may obtain animage of wound 1800 from a particular point of view corresponding to thefirst point in time by analyzing image data record 1900 of FIG. 19. Forinstance, server 145 may select one of images 1910, 1920, 1930, and/or1940 to generate a visual time series view of wound 1800.

In some embodiments, obtaining an image of a wound from a particularpoint of view corresponding to a first point in time may includeanalyzing images of the wound from other point of views corresponding tothe first point in time to generate the image of the wound from theparticular point of view corresponding to the first point in time.Similarly, obtaining an image of a wound from a particular point of viewcorresponding to a second point in time may include analyzing images ofthe wound from other points of view corresponding to the second point intime to generate the image of the wound from the particular point ofview corresponding to the second point in time. For example, a machinelearning model (for example, a generative model, such as generativeadversarial network, transformers based generative model, etc.) may betrained using training examples to generate images of desired points ofview based on images of other points of view. An example of suchtraining examples may include a sample image of a sample wound from asample point of view and an indication of the desired point of view,together with an image of the sample wound from the desired point ofview. The trained machine learning model may be used to analyze at leastone image of an image data record corresponding to one point in time(such as the first point in time or the second point in time) togenerate an image of the wound from the particular point of viewcorresponding to the that point in time. In some other examples, the oneor more images included in an image data record may include frames of avideo of the wound, the particular point of view may correspond to apoint of view in between two points of views corresponding to twoconsecutive frames of the video, and a video inpainting algorithm may beused to generate a new frame of the video between the two consecutiveframes and corresponding to the particular point of view. In some otherexamples, images of an image data record may be used to populate a 3Dtensor, where each specific image may populate a slice of the tensorcorresponding to a point of view associated with the specific image, andinterpolation algorithm may be used to complete a slice of the 3D tensorcorresponding to the particular point of view, therefore generating theimage of the wound from the particular point of view corresponding tothe point in time associated with the image data record.

Embodiments of the present disclosure may include generating a simulatedimage of the wound from the particular point of view corresponding tothe second point in time by analyzing the second image data record,wherein the second one or more images of the wound do not include animage of the wound from the particular point of view. In someembodiments, the second one or more images of the wound may not includean image of the wound from the particular point of view. In suchembodiments, a simulated image of the wound from the particular point ofview must be generated to be able to generate a visual time series viewof the wound including the same particular view throughout the two ormore included images. Analyzing the second image data record to generatethe simulated image of the wound from the particular point of viewcorresponding to the second point in time may refer to selecting one ormore images of the second one or more images of the wound which may besimilar to the image of the wound from the particular point of viewcorresponding to the first point in time and extracting data from theselected one or more images such that a simulated image of the wound atthe second point in time may be generated from the extracted data, thesimulated image of the wound corresponding to the particular point ofview. For instance, data may be extracted from the selected one or moreimages of the wound of the second one or more images of the wound suchthat a simulated image may be generated including a same particularillumination, viewing angle, orientation, image plane, distance,coloration, and/or other appropriate property as the image of the woundcorresponding to the first point in time. To this effect, both thesimulated image of the wound corresponding to the second point in timeand the image of the wound corresponding to the first point in time mayshare a particular point of view, even though no image of the second oneor more images shared the particular point of view of the image of thewound corresponding to the first point in time. In other examples, thesecond image data record may be analyzed to generate the simulated imageof the wound from the particular point of view corresponding to thesecond point in time as described above.

By way of example, a simulated image of wound 1800 from the particularpoint of view corresponding to the second point in time associated withimage data record 2000, as depicted in FIG. 20. In some embodiments,image data record 2000 may not include an image of wound 1800 from aparticular point of view. For instance, server 145 may select image 1920to generate visual time series view 2320 of FIG. 23, and image datarecord 2000 may not include an image of wound 1800 from the particularpoint of view of image 1920. In that circumstance, server 145 maygenerate a simulated image 2120 of wound 1800 from the particular pointof view corresponding to the second point in time by analyzing at leastimage 2020 of image data record 2000. Similarly, a simulated image 2110,as depicted in FIG. 21, from the particular point of view of image 1910may be generated by analyzing image 2010, and a simulated image 2130from the particular point of view of image 1930 may be generated byanalyzing image 2030. On the other hand, an image 2040 may already havethe particular point of view of image 1940 and may be included in imagedata record 2100 as-is or with minor modifications.

In some embodiments, the second image data record may include motiondata captured using an accelerometer associated with an image sensorused to capture the second one or more images of the wound, andanalyzing the second image data may include analyzing the motion data.Motion data may refer to information describing a motion of an imagesensor associated with the device used to capture the second one or moreimages of the wound included in the second image data record during thecapturing of the second one or more images of the wound. Anaccelerometer may refer to an instrument which may measure accelerationor motion. In some embodiments, analyzing the second image data recordto generate the simulated image of the wound may include analyzing themotion data. For instance, a motion or acceleration of the image sensorduring the capturing of the second one or more images of the wound maybe used to determine the point of view corresponding to different imagesincluded in the second image data record when generating the simulatedimage.

In some embodiments, generating the simulated image of the wound fromthe particular point of view corresponding to the second point in timemay include generating the simulated image to have selected illuminationcharacteristics. Illumination characteristics may include levels oflighting or light in an image. For example, the illuminationcharacteristics may be global for the entire image, or limited to aspecific region of the image (for example, to simulate shadows). In suchembodiments, generating the simulated image of the wound from theparticular point of view corresponding to the second point in time mayfurther include analyzing the image of the wound from the particularpoint of view corresponding to the first point in time to select theselected illumination characteristics. For instance, illuminationcharacteristics of the image of the wound from the particular point ofview corresponding to the first point in time may be determined toprovide the generated simulated image of the wound from the particularpoint of view corresponding to the second point in time with the same orsimilar illumination characteristics. By way of example, image 2010 ofFIG. 20 may have different illumination characteristics than image 1910of FIG. 19. As such, generating simulated image 2110 from the particularpoint of view of image 1910 may include further analyzing image 1910 todetermine an illumination characteristic and apply said illuminationcharacteristic to image 2010 to generate simulated image 2110, asdepicted in FIG. 21.

In some embodiments, the images of the wound from the particular pointof view corresponding to the first point in time and to the second pointin time may both correspond to a same distance from the wound. Forinstance, a distance of the image sensor to the wound in the image ofthe wound from the particular point of view corresponding to the firstpoint in time may be equal to a distance of the simulated image sensorto the wound in the simulated image of the wound from the particularpoint of view corresponding to the second point in time. Additionally,generating the simulated image of the wound from the particular point ofview corresponding to the second point in time may include generatingthe simulated image of the wound from the particular point of viewcorresponding to the second point in time by causing a distance from thewound in the simulated image to be equal to the distance from the woundassociated with the image of the wound from the particular point of viewcorresponding to the first point in time. That is, for example, adistance from the simulated image sensor to the wound in the simulatedimage of the wound from the particular point of view corresponding tothe second point in time may be modified to be equal to the distancefrom the image sensor to the wound in the image of the wound from theparticular point of view corresponding to the first point in time duringgeneration of the simulated image. By way of example, image 2030 of FIG.20 may be at a different distance to wound 1800 than image 1910 of FIG.19 is to wound 1800. As such, generating simulated image 2110 from theparticular point of view of image 1910 may include further analyzingimage 1910 to determine the distance to wound 1800 and may requiremodifying image 2030 to generate simulated image 2110, which may have asame distance from wound 1800 as image 1930. In some examples, aregression model may be used to analyze the image of the wound from theparticular point of view corresponding to the first point in time todetermine the distance from the wound associated with the image of thewound from the particular point of view corresponding to the first pointin time. In one example, a size of a wound in the simulated image may beselected and/or modified to correspond to the determined distance.

In some embodiments, the images of the wound from the particular pointof view corresponding to the first point in time and to the second pointin time both have a same spatial orientation. For instance, a spatialorientation of the image sensor with relation to the wound in the imageof the wound from the particular point of view corresponding to thefirst point in time may be equal to a spatial orientation of the imagesensor with relation to the wound in the simulated image of the woundfrom the particular point of view corresponding to the second point intime. In some embodiments, however, the spatial orientations may differ,for instance, if a visual time series view of a wound is intended toshow a wound from multiple angles. Additionally, generating thesimulated image of the wound from the particular point of viewcorresponding to the second point in time includes generating thesimulated image of the wound from the particular point of viewcorresponding to the second point in time by causing a spatialorientation in the simulated image to be equal to a spatial orientationassociated with the image of the wound from the particular point of viewcorresponding to the first point in time. That is, for example, aspatial orientation of the simulated image sensor in relation to thewound in the simulated image of the wound from the particular point ofview corresponding to the second point in time may be modified tocorrespond to the spatial orientation of the image sensor in relation tothe wound in the image of the wound from the particular point of viewcorresponding to the first point in time during generation of thesimulated image. By way of example, image 2020 of FIG. 20 may have adifferent spatial orientation than that of image 1920 of FIG. 19. Assuch, generating simulated image 2120 from the particular point of viewof image 1920 may include further analyzing image 1920 to determine thespatial orientation of image 1920 and applying said spatial orientationto image 2020 to generate simulated image 2120. In some examples, aregression model may be used to analyze the image of the wound from theparticular point of view corresponding to the first point in time todetermine the spatial orientation of the wound associated with the imageof the wound from the particular point of view corresponding to thefirst point in time. In one example, an affine transformation may beapplied to the wound in the simulated image to transform it tocorrespond to the determined spatial orientation.

In some embodiments, pixels of at least one matching pair of pixels ofthe image of the wound from the particular point of view correspondingto the first point in time and from the simulated image of the woundfrom the particular point of view corresponding to the second point intime correspond to a same physical length. That is, for example, whengenerating the simulated image of the wound from the particular point ofview corresponding to the second point in time, the simulated image maybe resized in order to match a set of pixels in the simulated image to asimilar set of pixels in the image of the wound from the particularpoint of view corresponding to the first point in time such that aphysical length in the simulated image corresponds to same physicallength in the image corresponding to the first point in time. By way ofexample, pixels of images 2200 and 2210, as depicted in FIG. 22 maycorrespond to a same physical length such that wound 1800 may maintain asame particular view throughout a visual time series view.

In some embodiments, each particular image of the wound from theparticular point of view corresponding to the first point in time and tothe second point in time include a visual indicator of a region of thewound corresponding to a particular tissue type in the particular image.Tissue types may include epithelial tissue, granulation tissue, sloughtissue, eschar, necrotic tissue, scab, hematoma, tendon, ligament, bone,infected tissue, non-infected tissue, or any other type of tissue whichmay be found in a wound. A visual indicator may include, for example,text, coloration, shading, or any other type of visual aid which maydifferentiate one region of a wound with another. In one example, asemantic segmentation algorithm may be used to analyze the images andidentify the region of the wound corresponding to the particular tissuetype. In some embodiments, each particular image of the wound from theparticular point of view corresponding to the first point in time and tothe second point in time may include a visual indicator of a depth ofthe wound at a particular location. A visual indicator of a depth of thewound may include, for example, dimensions, scales, or coloration. Byway of example, images 1920, 1930, 2020, 2030, 2120, and 2130 depict avisual indicator of regions of wound 1800 corresponding to particulartissue types. In one example, a regression algorithm may be used toanalyze the images and identify the depth of the wound at the particularlocation. Other algorithms for determining the depth of the wound whichmay be used are described herein.

Embodiments of the present disclosure may include generating a visualtime series view of the wound including at least the image of the woundfrom the particular point of view corresponding to the first point intime and the simulated image of the wound from the particular point ofview corresponding to the second point in time. A visual time seriesview of the wound may refer to a series of images, other visualrepresentations, and/or data relating to the wound ordered in time. Inone example, the visual time series of the wound may be a video of thewound including a frame depicting the wound from the particular point ofview corresponding to the first point in time and a frame depicting thewound from the particular point of view corresponding to the secondpoint in time. The visual time series view of the wound may include atleast the image of the wound from the particular point of viewcorresponding to the first point in time and the simulated image of thewound from the particular point of view corresponding to the secondpoint in time, ordered from the first point in time to the second pointin time, or from the second point in time to the first point in time.The visual time series view of the wound may include one or more imagescorresponding to a third point in time, a fourth point in time, and anyappropriate number of points in time. In some embodiments, each image ofthe images in the visual time series view of the wound may correspond toa point in time, and the images in the visual time series view of thewound may be ordered based on the corresponding points in time. By wayof example, visual time series views 2310, 2320, 2330, and 2340 includeimages of wound 1800 from a particular point of view corresponding to afirst point in time and a second point in time. For instance, visualtime series view 2310 includes image 1910 and simulated image 2110ordered based on corresponding points in time.

Consistent with some embodiments of the present disclosure, the imagesof the wound from the particular point of view corresponding to thefirst point in time and to the second point in time may both correspondto a same treatment phase of a treatment cycle of the wound. A treatmentcycle may refer to a series of steps that a wound may undertake duringtreatment. For instance, a wound may require daily treatment/cleaningand bandage changing, so a treatment cycle may include removing thebandages from the wound, cleaning and/or otherwise treating the wound,and applying new bandages to the wound. A treatment phase of thetreatment cycle of the wound may refer to a step of the treatment cycle.For instance, a treatment phase may refer to the removal of the bandagesfrom the wound, the cleaning and/or treatment of the wound, and theapplication of new bandages to the wound. The images of the wound fromthe particular point of view corresponding to the first point in timeand to the second point in time may therefore both correspond to thesame treatment phase of a treatment cycle of the wound. For example,both images may depict the wound before the bandages are removed, afterthe bandages are removed but before cleaning/treatment, aftercleaning/treatment, or after the new bandages are applied. In someembodiments, one or more pairs of images of the wound from theparticular point of view corresponding to the first point in time and tothe second point in time may be generated, wherein each pair of imagesof the wound correspond to a different treatment phase of a treatmentcycle of the wound. In some embodiments, the images of the wound fromthe particular point of view corresponding to the first point in timeand to the second point in time may both correspond to the wound beforeor after debridement, before or after dressing, or before or after anapplication of a medication to the wound. Debridement may refer to theremoval of nonviable material, foreign bodies, and poorly healing tissuefrom a wound. Dressing may refer to the application of bandaging to thewound and/or the area surrounding the wound.

By way of example, images of wound 1800 may correspond to a treatmentphase of a treatment cycle of wound 1800. For instance, images 1910,2010, and 2110 may correspond to wound 1800 before undressing, images1920, 2020, and 2120 may correspond to wound 1800 before debridement,images 1930, 2030, and 2130 may correspond to wound 1800 afterdebridement, and images 1940, 2040, and 2140 may correspond to wound1800 after dressing.

In some embodiments, generating the simulated image of the wound fromthe particular point of view corresponding to the second point in timemay include analyzing the image of the wound from the particular pointof view corresponding to the first point in time to determine atreatment phase of the treatment cycle of the wound corresponding to theimage of the wound from the particular point of view corresponding tothe first point in time and generating the simulated image of the woundfrom the particular point of view corresponding to the second point intime to correspond to the determined treatment phase. That is, forexample, the image of the wound from the particular point of viewcorresponding to the first point in time may be analyzed to ascertainwhich treatment phase of a treatment cycle the wound is in order togenerate a simulated image of the wound from the particular point ofview corresponding to the second point in time which matches thetreatment phase of the image of the wound corresponding to the firstpoint in time. In one example, a classification algorithm may be used toanalyze the image of the wound from the particular point of viewcorresponding to the first point in time to classify it to one of aplurality of alternative classes. Each class may correspond to atreatment phase of the treatment cycle of the wound, and theclassification of the image to the class may thereby determine thetreatment phase of the treatment cycle of the wound.

Some embodiments of the present disclosure may include calculating aconvolution of a part of an image of the first one or more images toderive a first result value, calculating a convolution of a part of animage of the second one or more images to derive a second result value,and determining a value of at least one pixel of the simulated image ofthe wound from the particular point of view corresponding to the secondpoint in time based on the first result value and the second resultvalue. In one example, the value of the at least one pixel of thesimulated image of the wound from the particular point of viewcorresponding to the second point in time may be a function of the firstresult value and the second result value. In another example, inresponse to one combination of the first result value and the secondresult value, a first value may be determined for the at least one pixelof the simulated image of the wound from the particular point of viewcorresponding to the second point in time, and in response to a secondcombination of the first result value and the second result value, asecond value may be determined for the at least one pixel of thesimulated image of the wound from the particular point of viewcorresponding to the second point in time.

Some embodiments of the present disclosure may include analyzing a firstimage of the first one or more images to detect a region of the woundcorresponding to a particular tissue type in the first image, analyzinga second image of the second one or more images to detect a region ofthe wound corresponding to the particular tissue type in the secondimage, and determining a value of at least one pixel of the simulatedimage of the wound from the particular point of view corresponding tothe second point in time based on the region of the wound correspondingto the particular tissue type in the first image and the region of thewound corresponding to the particular tissue type in the second image. Avalue of a pixel may refer to a coloration of a pixel, for instance, anRGB color value of a pixel. For example, a length of the detected regionof the wound corresponding to the particular tissue type in the secondimage may be used to determine a size of a region corresponding to theparticular tissue type in the simulated image, and a location of theregion of the wound corresponding to the particular tissue type in thefirst image may be used to determine a location of the regioncorresponding to the particular tissue type in the simulated image. Thelocation and size of the region corresponding to the particular tissuetype in the simulated image may be used to determine whether the atleast one pixel of the simulated image of the wound is in the regioncorresponding to the particular tissue type in the simulated image, andthe value of the at least one pixel may be determined based on whetherthe at least one pixel of the simulated image of the wound is in theregion corresponding to the particular tissue type in the simulatedimage.

FIG. 24 provides a flowchart of an example process 2400 for generatingvisual time series views of wounds including steps 2402 through 2408.Steps 2402 through 2408 may be executed by at least one processor (e.g.,processing device 202 of server 145 or mobile communications device 115of FIG. 2), consistent with some embodiments of the present disclosure.

Process 2400 may begin with step 2402. At step 2402, the at least oneprocessor may receive at least a first image data record and a secondimage data record, the first image data record corresponding to a firstpoint in time and including a first one or more images of a woundcaptured at the first point in time, and the second image data recordcorresponding to a second point in time and including a second one ormore images of the wound captured at the second point in time, theimages being captured by, for example, image sensor 226 of FIG. 2, whichmay be associated with a mobile device, such as communications device115.

Once the first and second image data records are received, process 2400may proceed to step 2404. At step 2404, the at least one processor mayobtain an image of the wound from a particular point of viewcorresponding to the first point in time by analyzing the first imagedata record.

At step 2406, the at least one processor may generate a simulated imageof the wound from the particular point of view corresponding to thesecond point in time by analyzing the second image data record, whereinthe second one or more images of the wound do not include an image ofthe wound from the particular point of view.

Once the simulated image has been generated, process 2400 may proceed tostep 2408. At step 2408, the at least one processor may generate avisual time series view of the wound including at least the image of thewound from the particular point of view corresponding to the first pointin time and the simulated image of the wound from the particular pointof view corresponding to the second point in time.

Embodiments consistent with the present disclosure provide systems,methods, devices, and computer readable media for rearranging andselecting frames of a medical video. For ease of discussion, in someinstances related embodiments are described below in connection with asystem or method with the understanding that the disclosed aspects ofthe system and method apply equally to each other as well as devices andcomputer readable media. Some aspects of a related method may occurelectronically over a network that is either wired, wireless, or both.Other aspects of such a method may occur using non-electronic means. Inthe broadest sense, the methods and computer readable media are notlimited to particular physical and/or electronic instrumentalities, butrather may be accomplished using many differing instrumentalities. Insome embodiments, the medical video may include a wound. A wound asreferred to herein may include any injury to the human body. Forexample, wounds may be open wounds resulting from penetration (e.g.,puncture wounds, surgical wounds and incisions, thermal, chemical, orelectric burns, bites and stings, gunshot wounds, etc.) and/or blunttrauma (e.g., abrasions, lacerations, skin tears), or they may includeclosed wounds (e.g., contusions, blisters, seromas, hematomas, crushinjuries, etc.). Some non-limiting examples of a wound may include achronic wound, acute wound, ulcer (such as venous ulcer, arterial ulcer,diabetic ulcer, pressure ulcer, etc.), infectious wound, ischemic wound,surgical wound, radiation poisoning wound, and so forth.

Disclosed embodiments may involve obtaining a desired property of asimulated trajectory of a virtual camera. A virtual camera as usedherein may refer to a camera that does not necessarily exist as aphysical camera, but is made by software to appear to do so. That is, avirtual camera may be computer-generated, and may be used, accessed, orstored by means of a computer and/or computer network (e.g., system 100in FIG. 1A and components thereof). For example, a virtual camera maynever exist, while a video may be generated to appear as if a virtualcamera with particular characteristics (such as position, orientation,motion, trajectory, zoom, focus, spectral sensitivity, focal length,field of view, resolution, color depth, frame rate, and so forth)captured the video. A trajectory may include any two dimensional orthree dimensional pathway between two or more points in physical space,and a simulated trajectory may include any two dimensional or threedimensional path between two or more points in a virtual space (e.g., asimulation of a physical space run by computer software). When referringto a simulated trajectory of a moving camera or a virtual camera, atrajectory may also include a viewing angle of the respective cameraalong the path of the respective trajectory. For example, in someembodiments, a trajectory of a moving camera includes a path followed bythe moving camera from a first position to a second position in physicalspace, and the simulated trajectory includes a generated path betweenthe first position and the second position, for example in acorresponding virtual space, in a physical space, and so forth. Thesimulated trajectory between the first position and the second positionmay be computer generated and may be configured to include a specificset of viewing angles of a wound.

Consistent with some embodiments of the present disclosure, thetrajectory of a moving camera and a corresponding simulated trajectorymay be different. For example, although a trajectory of a moving cameraand a trajectory of a counterpart virtual camera may both include thesame start position and end position, the path between the two positionsin the simulated trajectory, as well as the viewing angle of the virtualcamera along the path, may be different than that of the physicaltrajectory. For example, in some embodiments, the trajectory of themoving camera may include a diversion rendering at least a portion ofthe trajectory non-linear, and in one example the simulated trajectorydoes not include the diversion. Thus, the corresponding portion of thesimulated trajectory may differ from the trajectory of the movingcamera. In one example, the corresponding portion of the simulatedtrajectory may be linear, while the trajectory of the moving camera maybe non-linear. In another example, the corresponding portion of thesimulated trajectory may be smooth, while the trajectory of the movingcamera may be uneven. For example, the simulated trajectory may beconfigured to provide at least one view of a wound of a patient. Whilemoving along the simulated trajectory, the virtual camera may record orprovide a feed of a video of the wound. Consistent with disclosedembodiments, a simulated trajectory may include both linear andnon-linear portions. For example, in some embodiments, at least aportion of the simulated trajectory may be selected to be substantiallyon an arc of a circle, the wound being located at or near the center ofthe circle. The center of the circle may be positioned along the viewingangle of the virtual camera, such that the virtual camera is angleddirectly at or near the wound, consistent with some embodiments of thepresent disclosure.

In some embodiments, the simulated trajectory may be a standard woundviewing trajectory. That is, the trajectory of the virtual camera may beconfigured as to conform to a standard for capturing medical videosassociated with one or more healthcare providers, such as an associationof medical practitioners, a governing authority associated with thepractice of medicine, or any other entity associated with the provisionof healthcare (e.g., one or more individual hospitals, clinics, practiceareas, etc.). A standard consistent with the present disclosure may havespecific requirements for a particular property or range of propertiesthat a trajectory must conform with in order to comply with thestandard. For example, some non-limiting examples of propertiesassociated with standard wound viewing trajectories may include specificdirections, viewing angles, viewing distances, speeds, illuminationconditions, lengths, and the like, as discussed with further detailherein.

By way of example, FIGS. 25A and 25B provide a view of a wound 2500 onan arm of a patient in an X-Y plane and a Y-Z plane, respectively, andillustrate an example of a simulated trajectory of a virtual cameradelineated by double dashed lines 2550, consistent with some embodimentsof the present disclosure. For illustrative purposes and ease ofdiscussion, FIG. 25C provides yet another view of wound 2500 on the armof the patient in the X-Y plane, with virtual device 115(i) havingvirtual camera 226(i) illustrated translucently therein and beingpositioned along simulated trajectory 2550. Virtual camera 226(i) may bea virtual version of image sensor 226 of mobile communications device115 and may be simulated by one or more programs stored in at least onedata structure (e.g., memory device 234 of mobile communications device115 and/or database 146 of server 145 as illustrated in FIG. 2) whenexecuted by at least one processor (e.g., processing device 202 ofmobile communications device 115 and/or server 145) to capture a virtualvideo of wound 2500. Simulated trajectory 2550 may include a firstposition and a second position, the first position corresponding to aposition of image sensor 226 as it captures image 2510, and the secondposition corresponding to a position of image sensor 226 as it capturesimage 2530.

Virtual camera 226(i), as illustrated in FIG. 25C, may, in the firstposition, have a viewing angle directed to the radial portion of thepatient's forearm and, in the second position, have a viewing angledirected to the base of the patient's inner forearm. The viewing angleof virtual camera 226(i) may be maintained such that it remains directedat wound 2500 as it moves from the first position to the secondposition. FIGS. 25A and 25B also illustrate mobile communications device115 in a third position that is not included in simulated trajectory2550, in which image sensor 226 captures image 2520. This third positionmay correspond to a position of image sensor 226 that travels along atrajectory other than physical trajectory 2550 between capturing image2510 and 2530. Simulated trajectory 2550 may include linear andnon-linear portions. For example, at least one portion of simulatedtrajectory 2550 may include an arc of a circle being centered at or nearat least one portion of wound 2500. For example, an arc of trajectorymay be centered on a specific point on wound 2500, a boundary of wound2500, a contour of wound 2500, an axis tangentially aligned with aboundary of wound 2500, etc., or it may be centered around a specificpoint or axis near wound 2500, such as a calibrator (e.g., colorizedsurface 132 as illustrated in FIG. 4A) or around an axis coinciding withor parallel with the patent's forearm (e.g., the ulna or radius).

As discussed above, embodiments consistent with the present disclosuremay involve obtaining at least one desired property of the simulatedtrajectory. For example, the desired property of the simulatedtrajectory may be read from memory, may be received from an externaldevice, may be obtained from a user (for example, using a userinterface), may be determined automatically (for example, by analyzing avideo, for example using an egomotion algorithm, for example to mimic atrajectory associated with the video), and so forth. A property of thesimulated trajectory may include any physical or digital parameterassociated with the path of the trajectory (e.g., length, curvature,direction, etc.), the positioning and movement of the camera (e.g.,speed, direction, viewing angle, time of movement, etc.), configurationsof the camera and associated components (e.g., image resolution, framerate, gain, ISO speed, stereo base, lens, focus, zoom, color correctionprofile, flash, etc.), programs and programming configurations (e.g., inFIG. 2, sensor processing instructions 242, capturing instructions 254,application specific instructions 260, etc.) associated with a devicecapturing the video (e.g., image sensor 226), virtual device capturingthe virtual video (e.g., virtual camera 226(i)), and/or image processingdevice (e.g., processor 202 of communications device 115 or server 145,as illustrated in FIG. 2), and so forth. A specific set of propertiesmay be desired, for example, to improve the quality of the video, or tocomply with a particular standard associated with the video.

In some embodiments consistent with the present disclosure, the desiredproperty of the simulated trajectory of the virtual camera may include adesired moving direction of the virtual camera. Consistent with thepresent disclosure, the direction may be based on a standard thatrequires videos to be captured along a specific direction (e.g., left toright), or it may be based on physical properties of the patient and/orthe wound to be captured. For example, in some embodiments, obtainingthe desired property of the simulated trajectory may comprise selectingthe desired moving direction of the virtual camera based on a contour ofthe wound. In one example, the direction of the trajectory's path may beselected to align or closely align with a contour of the wound (e.g., aboundary of the wound or a boundary of a tissue type in the wound), suchthat the trajectory's path is configured such that the contour of thewound remains at least partially centered in the virtual camera's frameof view as the virtual camera moves along the simulated trajectory. Inanother example, the contour of the wound may be analyzed to determine alengthwise direction corresponding to the wound, and the desired movingdirection may be in the determined lengthwise direction, perpendicularto the determined lengthwise direction, in a selected angle with respectto the determined lengthwise direction, and so forth.

Consistent with some embodiments of the present disclosure, the desiredproperty of the simulated trajectory of the virtual camera may include adesired velocity of the virtual camera, and/or the desired property mayinclude a desired distance of the virtual camera from the wound. Thespeed of the virtual camera and the desired distance of the virtualcamera from the wound may be constant throughout the entire trajectory,or the speed and desired distance may vary based on any one or moreproperties associated with the wound, patient, camera, device, or soforth. For example, a lower speed and/or shorter distance may be desiredin order to capture more detailed images of a wound. In one example, theimage quality of one or more segments of the wound in particular (e.g.,a segment corresponding to a tissue type) may be desired. Thus, adistance and/or speed of the camera in at least one portion of thetrajectory corresponding with the one or more particular segments may bereduced to improve the quality and/or detail of the portion of thevirtual video containing the one or more particular segments. In otherexamples, a desired distance may be selected based on the focal lengthof the camera or due to the dimensions of the wound. In yet anotherexample, the velocity of the virtual camera may be selected based on thetime dependent properties of the camera (e.g., frame rate, gain, ISOspeed, etc.) or on the dimensions of the wound. Some additionalnon-limiting examples of factors affecting the desired velocity of thevirtual camera and/or desired distance of the virtual camera from thewound may include illumination conditions (for example, having lowervelocity and/or shorter distance when the illumination conditions arepoor), condition of the wound, tissue composition of the wound, depth ofthe wound, and so forth.

By way of example, in FIGS. 25A-C, wound 2500 has a snake-like shapethat has one end located near the radial side of the patient's wrist andanother end located on the base of the patient's inner forearm. In someembodiments, the direction of movement of virtual camera 226(i), asillustrated in FIG. 25C, may be selected to roughly follow (or toprecisely follow, in some embodiments) the contour of wound 2500, suchthat the virtual video captured by virtual camera 226(i) includes imagesof wound 2500. In some embodiments, virtual camera 226(i) may travel atvariable velocity and at a variable distance from wound 2500 alongtrajectory 2550, consistent with some embodiments with the presentdisclosure. For example, wound 2500 may include segments 2500(a)-(c),where a high amount of detail corresponding to the video of segment2500(b) is desired. The velocity of virtual camera 226(i) may be reducedas segment 2500(b) passes through the frame of the virtual imagecaptured by virtual camera 226(i) as virtual camera 226(i) moves throughthe corresponding portion of simulated trajectory 2550. Thecorresponding portion of simulated trajectory 2550 may, in someembodiments, be associated with a shorter distance from wound 2550 inorder for virtual camera 226(i) to collect images with more detail.

Some embodiments consistent with the present disclosure may includeanalyzing at least one image of the wound to determine a condition of atleast part of the wound. Consistent with the present disclosure, acondition of a wound as referred to herein may refer to a medicalcondition (e.g., infection), healing stage, or any other physicalparameter associated with a wound. A condition associated with a woundmay, for example, be determined by a medical professional (e.g., medicalpractitioner 120 in FIG. 1A) and placed into a record corresponding tothe wound (e.g., saved in database 146). However, some embodiments ofthe disclosure may include using machine learning, as previouslydiscussed herein, to determine a condition of a wound or to estimateand/or interpolate a condition of a wound. For example, in someembodiments, a machine learning model (e.g., a classification model) maybe trained using training examples to determine one or more conditionsof a wound in one or more images. Examples of training examples fordetermining a condition of a wound may include sample images of woundshaving known conditions (e.g., an infected wound with predeterminedmeasurements, color, tissue types, etc.). The trained machine learningmodel may be used to analyze the at least one image of the wound todetermine a condition of the wound. In some embodiments, the at leastone analyzed image may be an image in the captured video, one or moreseparate captured videos, one or more separately captured images, or oneor more images in a virtual video.

In some embodiments, the simulated trajectory of the virtual camera maybe determined based on a condition of the at least part of the wound.The condition of the at least part of the wound may correspond to aparticular segment of the wound, or the condition itself may constitutea segment of a wound. Determining the simulated trajectory based on thecondition of the wound may, for example, involve obtaining one or moredesired properties of a simulated trajectory configured to capture avideo containing quality image data associated with the condition. Insome embodiments, the simulated trajectory may be configured tocorrespond with a physical parameter and/or dimension of the specificcondition. For example, the desired properties of the trajectory mayinclude one or more directions and/or distances configured to enable thevirtual camera to capture a video containing images of the at least aportion of the wound with the condition. In some embodiments, the atleast one simulated trajectory may be determined based on acharacteristic of the condition (e.g., a type of infection). In oneexample, a wound may be infected, and at least a portion of the desiredtrajectory may involve a low speed and/or distance of the virtual cameraas it moves along one or more directions configured to roughly follow acontour of the infected segment of the wound. By way of example, segment2500(b) may be an infected portion of wound 2500, and a specific desiredvelocity of virtual camera 226(i) and distance of camera 226(i) fromsegment 2500(b) may be selected in order to obtain a virtual video ofsegment 2500(b) with a high amount of detail.

Some embodiments consistent with the present disclosure may includeanalyzing at least one image of the wound to identify a first region ofthe wound corresponding to a first tissue type and a second region ofthe wound corresponding to a second tissue type. Identifying a region ofthe wound based on a tissue type may refer to receiving, generating, orotherwise acquiring a division into separate parts or regions of thewound based on different tissue types present in the wound. Forinstance, a wound may be segmented into regions based on different areasof the wound consisting of different types of tissues. Tissue types mayinclude epithelial tissue, granulation tissue, slough tissue, eschar,necrotic tissue, scab, hematoma, tendon, ligament, bone, infectedtissue, non-infected tissue, or any other type of tissue which may befound in a wound. For example, in some embodiments, a machine learningmodel (e.g., a semantic segmentation model, etc.) may be trained usingtraining examples to identify one or more regions of a woundcorresponding to one or more tissue types. Examples of training examplesfor a tissue type may include sample images of wounds having knowntissue types (e.g., images of tendons, ligaments, bones, etc.). Thetrained machine learning model may be used to analyze the at least oneimage of the wound to identify at least one region of the woundcorresponding to a particular tissue type. As discussed above, the atleast one analyzed image may be an image in the captured video, one ormore separate captured videos, one or more separately captured images,or one or more images in a virtual video.

In some embodiments, the simulated trajectory of the virtual camera maybe determined based on a dimension of the first region of the wound, thefirst tissue type, a dimension of the second region of the wound, andthe second tissue type. Determining the simulated trajectory based onthe dimensions of a tissue type may include, for example, obtaining oneor more desired properties of a simulated trajectory configured tocapture a video containing images of at least a portion of the firstregion and/or the second region. In some embodiments, for example, thesimulated trajectory may be configured to capture at least one imagewith the entire dimension or at least a portion of the dimension in theframe. For example, in some embodiments, a constant number of frames maybe allocated for the new video of the wound (for example, to keep thelength of the video fixed). To allocate the frames to a first portion ofthe simulated trajectory associated with the first region of the woundand to a second portion of the simulated trajectory associated with thesecond region of the wound, a weight for each portion may be calculatedbased on the dimension of the region and the tissue type correspondingto the region associated with the portion, and a ratio of the constantnumber of frames proportional to the weight corresponding to the portionmay be allocated to the portion. In one example, each tissue type maycorrespond to a predetermined factor, and the weight corresponding to aportion of the simulated trajectory may be a multiplication of thecorresponding factor and dimension. In another example, a numericalparameter may be determined based on the tissue type, and the weightcorresponding to a portion of the simulated trajectory may be calculatedusing a parametric function of the corresponding dimension using thedetermined numerical parameter. By way of example, in FIG. 25A-C, wound2500 may include tissue segments 2500(a)-(c). In some embodiments,segments 2500(a) and 2500(c) may correspond to a first tissue type, andsegment 2500(b) may correspond to a second tissue type. In one example,segments 2500(a) and 2500(c) may be made of scab tissue, whereas segment2500(b) may be exposed epithelial tissue. If, for example, a videoexamining the epithelial tissue of segment 2500(b) is desired, thensimulated trajectory 2550 may be specifically configured to capture ahigh quality video of segment 2500(b) using the dimensions of segments2500(a)-2500(b).

Embodiments consistent with the present disclosure may involve receivinga first video of a wound captured by a moving camera. The first videomay include a plurality of frames. Consistent with some embodiments ofthe present disclosure, the plurality of frames may include at least twoframes corresponding to the simulated trajectory of the virtual camera.In some embodiments, the first video may be analyzed to select the atleast two frames corresponding to the simulated trajectory of thevirtual camera using the at least one desired property of the simulatedtrajectory. In some examples, embodiments of the present disclosure mayinvolve determining at least one property associated with the pluralityof frames in the first video (e.g., using image processing as discussedpreviously herein). The at least one property of the plurality of imageframes may be compared with at least one desired property of thesimulated trajectory to determine that the at least two framescorrespond to the simulated trajectory. By way of example FIG. 25A, thefirst video may be a video captured by image sensor 226 of mobilecommunications device 115. The trajectory of image sensor 226 mayinclude the position where image sensor 226 captures image 2510, anotherposition where image sensor 226 captures image 2520, and yet anotherposition where image sensor 226 captures image 2530. The video capturedby image sensor 226 may include frames including image 2510, image 2520,and image 2530. In some embodiments, at least one processor (e.g.,processor 202 of mobile communications device 115 or server 145, asillustrated in FIG. 2) may be configured to perform image processing onimages 2510, 2520, and 2530 (e.g., by calculating a convolution toderive a result value) to determine certain properties of images 2510,2520, and 2530. Based on a comparison between the determined propertiesand the desired properties of trajectory 2550, the at least oneprocessor may determine that images 2510 and 2530 correspond tosimulated trajectory 2550.

In some disclosed embodiments, the simulated trajectory may be selectedbased on a second video of a wound captured at a different time. Forexample, in some embodiments, the simulated trajectory may be configuredto follow a similar trajectory that was previously used to capture avideo of the wound. In this sense, the simulated trajectory of thevirtual camera may appear to be a recreation of the previously capturedvideo. In some embodiments, the wound in the second video captured at adifferent time may be the same wound as the wound in the first video.However, in some embodiments, the two wounds may be different, withenough similar features such that the trajectory of the moving camerafrom the second video is desired to capture a virtual video using framesfrom the first video (e.g., similar patients, similar injuries, similarlimbs, etc.). Some disclosed embodiments may also include causing adisplay of the second video in conjunction with a display of a new videocreated with the simulated trajectory, as discussed in further detailbelow. The second video may be displayed, for example, alongside aplayback of the new video as to provide a user with a comparison view ofthe wound in the first video and the wound in the second video, forexample in a user interface. In another example, the second video may bedisplayed as an overlay over a playback of the new video.

By way of example, in FIGS. 25A-C, as previously discussed herein, afirst video may be obtained using image sensor 226 of mobilecommunications device 115. Simulated trajectory 2550 may be determined,for example, by obtaining one or more desired properties of trajectory2550. In some, embodiments, however, simulated trajectory 2550 may bebased on the trajectory of a camera in a previously captured video. Thatis, simulated trajectory may be based on a video recorded alongtrajectory 2550, the second video including images taken in the sameposition as images 2510 and 2530.

Some embodiments may include using the desired property of the simulatedtrajectory of the virtual camera to select an order for the selected atleast two frames. In some examples, the order of the selected at leasttwo frames may selected in the same order that they were captured in thefirst video. For example, in FIGS. 25A-C, assuming the at least twoframes include images 2510 and 2530, frames 2510 and 2530 may beselected in the order they were captured. However, in some embodiments,images 2510 and 2530 may be selected in the opposite order, for exampleif the desired direction of trajectory 2550 is in a direction that isopposing the direction traveled by image sensor 226 or if frames 2510and 2530 are otherwise captured in a different order then they are toappear in the virtual video along simulated trajectory 2550. In otherexamples, when the selected at least two frames are at least threeframes, the selected order may include any possible rearrangement of theat least three frames. For example, a middle frame of the at least threeframes (in the order of capturing of the frames) may be the first framein the selected order, may be the last frame in the selected order, maybe a middle frame in the selected order, and so forth. Likewise, a firstframe or a last frame of the at least three frames (in the order ofcapturing of the frames) may be the first frame in the selected order,may be the last frame in the selected order, may be a middle frame inthe selected order, and so forth.

Embodiments consistent with the present disclosure may includerearranging the at least two frames based on the selected order tocreate a new video of the wound that represents the simulated trajectoryof the virtual camera. A new video reflecting the simulated trajectoryof the virtual camera may include a video that, as viewed by a user, mayappear as having been captured by an actual camera that captured the atleast two frames along the simulated trajectory. As discussed above, theselected order of the at least two frames may be any order, regardlessof which order the frames were captured in, in accordance with thedesired properties of the simulated trajectory. Thus, the new videocreated to reflect the simulated trajectory may include any number ofselected frames from an actual video arranged in any order, so long asthe selected frames include the desired properties of the simulatedtrajectory.

In some embodiments consistent with the present disclosure, creating thenew video comprises generating at least one synthetic frame by analyzingthe first video, and wherein the new video includes the at least onesynthetic frame. Generating a synthetic frame may involve interpolation,where the frame is synthesized in between existing frames, or it mayinvolve extrapolation, where the frame is synthesized subsequent toexisting frames. In some embodiments, a machine learning model (e.g., agenerative model, such as a generative adversarial network, atransformers-based model, etc.) may be trained using training examplesto generate synthetic frames. Examples of training examples may includeany video including a plurality of frames, as the model can use aportion of the frames in any video as a training data set with at leasta portion of the remaining frames as desired frames for generation.Specifically, a machine learning model for generating synthetic framesof a medical video may, for example, be trained using any number ofvideos of wounds. The trained machine learning model may be used toanalyze the first video. By way of example, in FIGS. 25A-C, the firstvideo may include frames 2510 and 2530, as previously discussed. In someembodiments, at least one processor (e.g., processor 202 of mobilecommunications device 115 or device 145) may implement a machinelearning model in analyzing the first video including frames 2510 and2530 to generate a new frame, for example frame 2520(i). In anotherexample, one or more synthetic frames may be generated to correspond toa particular point of view, for example as described herein.

In some embodiments consistent with the present disclosure, creating anew video may include modifying frames of the first video using at leastone correction factor. For example, in some embodiments, creating a newvideo may include receiving a first correction factor associated with afirst portion of the new video of the wound and second correction factorassociated with a second portion of the new video of the wound, andwherein creating the new video of the wound includes modifying frames ofthe first portion of the new video of the wound based on the firstcorrection factor and modifying frames of the second portion of the newvideo of the wound based on the second correction factor. A correctionfactor may include a factor that may be applied to a given output tocorrect for a known amount of error, for differing illuminationconditions, for differing distances from the wound, for differing sizesof wounds, and so forth. In some disclosed embodiments, some of theproperties of the desired result video may be known. For example, insome embodiments, machine learning models may be trained with trainingexamples of medical videos to determine a known amount of error betweenproperties of a result new video and a desired new video.

For example, in some embodiments, the amount of error between resultillumination conditions of the new video and illumination of desiredillumination conditions of a new video may be known. Accordingly, insome embodiments, a first correction factor corresponds to a firstillumination condition and the second correction factor corresponds to asecond illumination condition. By applying the correction factors to thenew video, the desired illumination conditions for the new video may beachieved. In another example, the amount of error associated with aresulting distance of at least camera from a wound in a created newvideo may be known. Accordingly, in some embodiments, a first correctionfactor corresponds to a first distance from the wound and a known secondcorrection factor corresponds to a second distance from the wound. Byapplying the correction factors to the first video, the desired distanceof the camera from the wound as it appears in the created new video maybe achieved.

As previously discussed, some embodiments may include analyzing at leastone image of the wound to identify a first region of the woundcorresponding to a first tissue type and a second region of the woundcorresponding to a second tissue type. For instance, a wound may besegmented into regions based on different areas of the wound consistingof different types of tissues, thereby distinguishing between differentportions of the wound with different tissue types (for example, using asemantic segmentation algorithm). To be clear, some embodiments include,determining the first portion of the new video of the wound based on thefirst region of the wound and a second portion of the new video of thewound based on the second region of the wound and determining the firstcorrection factor based on the first tissue type and the secondcorrection factor based on the second tissue type. In some embodiments,correction factors may be determined based on the same or differenttissue types. In one example, each tissue type may be associated with apredetermined correction factor. In another example, a correction factorfor a portion of the new video corresponding to a region of the woundand a tissue type may be determined based on a function of the tissuetype and at least one additional parameter (such as a dimension of theregion of the wound, a tissue type of another region of the woundadjacent to the region of the wound, an illumination condition, and soforth).

FIG. 26 is a flowchart of an example process 2600 for rearranging andselecting frames of medical videos including steps 2602 through 2610.Steps 2602 through 2610 may be executed by at least one processor (e.g.,processing device 202 of server 145 or mobile communications device 115of FIG. 2), consistent with some embodiments of the present disclosure.

Process 2600 may begin with step 2602. At step 2602, the at least oneprocessor may obtain a desired property of a simulated trajectory (e.g.,simulated trajectory 2550 in FIGS. 25A-C) of a virtual camera (e.g.,virtual camera 226(i) of virtual mobile communications device 226(i) inFIG. 25C).

Once the desired property has been obtained, process 2600 may proceed tostep 2604. At step 2604, the at least one processor may receiving afirst video of a wound (e.g., wound 2500) captured by a moving camera(e.g., image sensor 226), the first video including a plurality offrames (e.g., frames 2510, 2520, 2530).

After the first video including a plurality of frames has been received,process 2604 may proceed to step 2606. At step 2606, the at least onprocessor may use the desired property of the simulated trajectory ofthe virtual camera to analyze the first video to select at least twoframes (e.g., frames 2510 and 2530) of the plurality of framescorresponding to the simulated trajectory of the virtual camera. At step2608, the at least one processor may further use the desired property ofthe simulated trajectory of the virtual camera to select an order forthe selected at least two frames. Finally, at step 2610, the at leastone processor may rearrange the at least two frames based on theselected order to create a new video of the wound that represents thesimulated trajectory of the virtual camera.

Embodiments consistent with the present disclosure provide systems,methods, devices, and computer readable media storing instructions forcapturing and analyzing images to providing wound capturing guidance. Inone example, consistent with the disclosed embodiments, an exemplarysystem may receive one or more images depicting a wound or other tissuefeature from at least one image sensor. A “wound” as referred to hereinmay include any injury to the human body. For example, wounds may beopen wounds resulting from penetration (e.g., puncture wounds, surgicalwounds and incisions, thermal, chemical, or electric burns, bites andstings, gunshot wounds, etc.) and/or blunt trauma (e.g., abrasions,lacerations, skin tears), or they may include closed wounds (e.g.,contusions, blisters, seromas, hematomas, crush injuries, etc.). Somenon-limiting examples of a wound may include a chronic wound, acutewounds, ulcer (such as venous ulcer, arterial ulcer, diabetic ulcer,pressure ulcer, etc.), infectious wound, ischemic wound, surgical wound,radiation poisoning wound, and so forth. Based on an analysis of theimages, the exemplary system may provide, through a user interface,guidance to place the wound in a desired position in the imaging frameand/or move a device associated with the at least one image sensor in adesired direction and/or to rotate the device in a desired way. By wayof example, in FIG. 27, communications device 115 may be configured witha user interface that may guide the user to capture additional desiredimages of wound 2700 by repositioning wound 2700 in the image frame orto move communications device 115 in a desired direction based on ananalysis of one or more original images captured by image sensor 226(not shown in FIG. 27) and/or motion data provided by motion sensor 228(not shown in FIG. 27) in communications device 115.

Embodiments consistent with the present disclosure may include receivinga plurality of frames from at least one image sensor associated with amobile device, at least one of the plurality of frames containing animage of a wound. A plurality of images may refer to multiple individualimages captured individually at different times, or it may refer to aplurality of images captured as a continuous video feed. By way ofexample, in FIG. 27, a plurality of images captured by communicationsdevice 115 may be received (e.g., the image displayed by communicationsdevice 115 as illustrated in FIG. 27, or the images in FIG. 29 displayedon mobile communication device 115 in the positions denoted as115(1)-(3)). At least one of the images may include wound 2700 or aportion thereof.

Embodiments consistent with the present disclosure may includedisplaying, on the mobile device, a real time video including at least aportion of the plurality of frames and a visual overlay indicating adesired position of the wound. For example, a user interface on a mobilecommunications device may display a live video feed captured from atleast one image sensor of the device. On the user interface, the devicemay also display a visual indication of a position on the video feed atwhich the image of the wound should be positioned. For example, in someembodiments, the visual overlay may include an indication of a desiredposition for a center of the wound, or the visual overlay may include anindication of a bounding shape for the wound in the image or video. Theindication of the desired position for the center of the wound may be inthe form of crosshairs, pointed arrows, a dot, or any other visualelement appropriate for designating a desired position on a display. Thebounding shape for the wound may be a simple shape, such as a circle,square, or other polygon, or it may be a shape that is generated toclosely resemble the shape of the wound.

By way of some non-limiting examples, FIGS. 28A and 28B illustratecommunication devices 115A and 115B displaying images of wounds 2802Aand 2802B, respectively, and indications of desired positions thereof.For example, in FIG. 28A, the overlay of communications device 115includes an indication 2804 of the center of wound 2802A and crosshairs2806 indicating a desired position of the center of the wound. In FIG.28B, the overlay of communications device 115 includes an indication ofa bounding shape 2808 of wound 2802B indicating the desired position ofthe wound. Furthermore, although the term “indication” is typically usedherein as being displayed on a mobile device, this descriptive use isfor illustrative purposes only. For example, indications may alsoinclude audible and/or tactile indications and may be provided onsystems other than a mobile device (e.g., through external monitors,speakers, augmented reality systems, virtual reality systems, etc.).Thus, it is to be understood that the foregoing illustrativedescriptions are not meant to limit the present disclosure to certainembodiments that utilize a physical display to provide indications.

Embodiments consistent with the present disclosure may includedetecting, based on at least part of the plurality of frames, that thewound is in the desired position. For example, the at least part of theplurality of frames used for the detection may be identical to thedisplayed at least a portion of the plurality of frames or may differfrom the displayed at least a portion of the plurality of frames. Forexample, the at least part of the plurality of frames used for thedetection and the displayed at least a portion of the plurality offrames may have all frames in common, may have no frames in common, mayhave some but not all frames in common, and so forth. In some examples,a mobile communications device imaging the wound may perform an analysison the captured images to determine whether the wound is in a desiredposition, or it may determine that it is in a desired position based onthe relative position of the wound in the overlay of the mobilecommunications device with respect to an indication of the desiredposition. For example, a machine learning model may be trained usingtraining examples to determine whether the wounds are in desiredpositions in image frames. An example of such training example mayinclude a sample image of a sample wound and an indication of a sampledesired position for the sample wound in the sample image, together witha label indicating whether the sample wound is in the sample desiredposition. The trained machine learning model may be used to analyze theat least part of the plurality of frames to detect that the wound is inthe desired position or that the wound is not in the desired position.In other examples, object detection algorithm may be used to analyze theat least part of the plurality of frames to determine an actual positionof the wound in the at least part of the plurality of frames, and theactual position may be compared with the desired position to detect thatthe wound is in the desired position or that the wound is not in thedesired position. If the wound is not in the desired position,embodiments consistent with the present disclosure may includedisplaying an indication to correct an actual position of the wound inthe video. Once the actual position of the wound is corrected, thedisplay of the indication to correct the actual position of the wound inthe video may be halted, or an additional indication indicating that theposition has been corrected may be displayed.

By way of some non-limiting examples, FIGS. 28A and 28B illustratecommunication devices 115A and 115B displaying images of wounds 2802Aand 2802B in incorrect positions and displaying indications to correctthe actual positions of the wounds. For example, in FIG. 28A,communications device 115A may detect that wound 2802A is not in thedesired position because the center 2804 of wound 2802A is not alignedwith crosshairs 2806. In response, communications device 115A maydisplay indication 2810A, prompting the user “to move the indicatedcenter of the wound into the crosshairs.” Once center 2804 has beenaligned with the crosshairs 2806, communications device 115A may removeindication 2810A or replace it with an indication that wound 2802A is inthe desired position. In FIG. 28B, communications device 115B may detectthat wound 2802B is not in the desired position because it does notcoincide with the bounded shape of the wound 2808. In response,communications device 115B may display indication 2810B, prompting theuser to “move the wound into the indicated area.” Once wound 2802Bcoincides with bounded area 2808, communications device 115B may removeindication 2810B or replace it with an indication that wound 2802B is inthe desired position.

In some embodiments, image analysis may include calculating aconvolution of the at least part of the plurality of frames to derive aresult value of the calculated convolution, for example as describedabove. In some embodiments, the derived result value of the calculatedconvolution may be used to determine an actual position of the wound. Inone example, in response to a first result value of the calculatedconvolution, a first actual position of the wound may be determined, andin response to a second result value of the calculated convolution, asecond actual position of the wound may be determined, the secondposition may differ from the first position. In another example, thedetermined actual position of the wound may be a function of the resultvalue of the calculated convolution. Some non-limiting examples of suchfunction may include a linear function, a non-linear function, apolynomial function, a logarithmic function, an exponential function, acontinuous function, a non-continuous function, a monotonic function, anon-monotonic function, and so forth. In yet another example, thederived result value may be used to determine a position of a certainelement of the wound (e.g., center 2804 of wound 2802A in the imagedisplayed on communications device 115A) or the edge or boundary of awound (e.g., the boundary of wound 2802B in the image displayed oncommunications device 115B), or any other position associated with thewound. Once the actual position of the wound has been determined, theactual position may be compared with the desired position of the wound(e.g., the positions indicated by crosshairs 2806 or boundary 2808) todetect that the wound is in the desired position. In some embodiments,this detection may be based on the difference between the actualposition of the wound and the desired position being greater than athreshold value (e.g., by a number of pixels or other coordinatevalues).

Some embodiments may include detecting that the wound is not in thedesired position for at least a specified period of time and, inresponse, displaying an indication to correct an actual position of thewound in the video. For example, when it is determined that a wound isnot in a desired position within a frame based on an analysis of atleast a part of the plurality of frames captured, an indication tocorrect the actual position of wound may not be displayed until it hasnot been in the desired position for a predetermined amount of time(e.g., 0.5 s, 1 s, 2 s, 5 s, etc.). By way of example, in FIG. 28B,indication 2810B may be displayed if the actual position of 2802B on thedisplay of mobile communications device 115B does not coincide withbounded area 2808 for two seconds. Prior to the expiration of those twoseconds, indication 2810B may not be displayed.

Embodiments consistent with the present disclosure may includedisplaying an indication on the mobile device to move the mobile devicein a desired direction. For example, in some situations, the size,shape, and/or positioning of the wound may require a user to makeadditional movements with the mobile device to capture additional imagesin order to collect as much information from the imaged wound aspossible. In some non-limiting examples, the displayed indication may betextual, graphical, a combination of a text with graphics, and so forth.In another non-limiting example, the indication may be provided audibly.By way of example, in FIGS. 27 and 29, the patient is inflicted withwound 2700 extending from the base of the patient's forearm to theradial side of the patient's wrist. In such situations, a single imageof the wound may not provide sufficient information for a computerizedsystem or a physician to make an effective evaluation of the wound'scondition. For example, an image taken directly above and perpendicularto the posterior portion of a forearm may not provide much valuableinformation regarding the portion of the wound on a radial side of thepatient's wrist, and no information whatsoever regarding any portions ofthe wound on the anterior side of the patient's forearm. Thus, it may benecessary to capture a series of images to capture the entirety of thewound. In some embodiments, once a portion of the wound is in thedesired position on the user interface of the mobile device, the mobiledevice may display an indication to prompt the user to move the devicein order to capture additional images. The displayed indication mayinclude directions to move the mobile device in one or more directionsin three dimensions and/or rotate the mobile device about one or moreaxes. In general, a “desired direction” may include any directional orrotational trajectory.

By way of example, FIG. 29 illustrates the capturing of a plurality ofimages of wound 2700 by moving communications device 115. Notation mark115(1) denotes communications device 115 in a first position, in whichthe user has moved the actual position of the wound to the initialdesired position on the user interface. In response to the actualposition of the wound being in the desired position, communicationsdevice 115 may display indication 2910 prompting the user to “move thedevice in the indicated direction” (or the like) and a direction arrow2902 directing the user to the desired rotation. Although the directionarrow 2902 is illustrated as two-dimensional herein, it is to beunderstood that the provided direction can be configured to appearthree-dimensional, consistent with disclosed embodiments and at leastthe capabilities of conventional display and computation devices. Forexample, direction arrow 2902 and indication 2910 can be configured toprompt the user to rotate the mobile device along one or more axes inaddition to prompting the user to move the device in one or moredirections.

Embodiments consistent with the present disclosure may include receivingmotion data from at least one motion sensor associated with the mobiledevice. Motion sensors (e.g., motion sensor 228 depicted in FIG. 2) mayinclude accelerometers, gyroscopes, or any other sensor configured tomeasure acceleration, gravity, speed of revolution, curl vector values,or drift of the mobile device. In some embodiments, the motion may bedetermined, at least in part, based on an analysis of the plurality ofimages captured by the at least one image sensor (e.g., image sensor 226depicted in FIG. 2) of the mobile device, for example by analyzing theplurality of images with an egomotion algorithm. In this sense, an imagesensor may also be considered to be a motion sensor.

Based on the received motion data, embodiments consistent with thepresent disclosure may include detecting that the mobile device hasmoved in the desired direction. For example, in some embodiments, adirection of the actual movement of the mobile device may be determinedat least in part on motion data received from the at least one motionsensor, and the direction of the actual movement may then be comparedwith the desired direction to determine whether the mobile device ismoving in the desired direction. In some embodiments, this determinationmay be made upon the direction of the actual movement the mobile deviceexceeds a predetermined tolerance. That is, the desired movement may beconsidered to include a range of directions, such that the mobile devicemay be determined to be moving in the desired direction if the directionof actual movement is within a given range of angles from a desireddirection.

By way of example, in FIG. 29, communications device 115 may include atleast one motion sensor (e.g., image sensor 226 and motion sensor 228,not shown in FIG. 29) that may generate motion data that can be utilizedto determine the direction of actual movement of communications device115 as a user moves its position according to indication 2910 and thedirection arrow 2902. When communications device 115 has moved in thecorrect direction (e.g., by moving to the position denoted with 115(3)),it may be determined that communications device 115 has moved in thedesired direction because it has moved in a direction within apredetermined tolerance (e.g., within 1 degree, 5 degrees, 15 degrees,30 degrees, etc.) of the indicated desired direction indicated by arrow2902.

Some embodiments consistent with disclosed embodiments may includedetecting that the mobile device has moved in a direction different fromthe desired direction and, in response, displaying an indication on themobile device to correct the movement of the mobile device. For example,as described above, the motion data collected by the one or more motionsensors of the mobile device may be used to determine the direction ofactual movement. If the actual direction of movement is not the same as(or exceeds a predetermined tolerance of) the desired direction, it maybe accurately determined that the mobile device has moved in anincorrect direction, and the mobile device can accordingly provide anindication to correct the movement of the mobile device. By way ofexample, in FIG. 29, when communications device 115 has moved in adifferent direction than the desired direction (e.g., by moving to theposition denoted with 115(2), where the wound is no longer in thecollected image, or when mobile device 115 has moved in a directiondiffering from desired direction 2902 in excess of a predeterminedtolerance), communications device 115 may be configured to display anindication 2912 directing the user to move communications device 115 indirection 2904, thereby correcting the movement of communications device115(2). In some non-limiting examples, the displayed indication may betextual, graphical, a combination of a text with graphics, and so forth.In another non-limiting example, the indication may be provided audibly.

Embodiments consistent with the present disclosure may includedisplaying an additional indication on the mobile device when the mobiledevice has moved in the desired direction. In some embodiments, theindication may be that the needed images have been captured or thatimage capturing of the wound has been completed. Alternatively, in someembodiments, additional imaging of the wound in one or more differentdirections may be required to collect all of the necessary information(e.g., where the wound is elongated and extends in multiple longitudinaldirections, where the wound cannot be entirely imaged by moving themobile device in one direction due to size, etc.). Accordingly, in someembodiments, the additional indication may include an instruction tomove the mobile device in a different direction. In some embodiments,once all imaging of the wound has been completed due to the mobiledevice being moved in one or more desired directions (e.g., along adesired trajectory), the captured image data and motion data may be usedto construct a three-dimensional model of the wound. Further, someembodiments may include generating a user rating based on an analysis ofat least one frame of the plurality of frames. The user rating mayinclude a score, a percentage, or any other metric indicative of theuser's actual positioning and/or movement of the mobile device relativeto the desired positions and/or directions indicated by the mobiledevice. The user rating may, for example, be based on a comparisonbetween the actual positions and movements of the mobile device and thedesired positions and movements of the mobile device. In somenon-limiting examples, the additional indication may be textual,graphical, a combination of a text with graphics, and so forth. Inanother non-limiting example, the additional indication may be providedaudibly.

By way of example, in FIG. 29, once communications device 115 has movedin a direction consistent with a desired direction indicated by arrow2902 (e.g., by moving to the position denoted with 115(3)),communications device 115 may display an indication 2914 notifying theuser that imaging has been completed. Alternatively, if imaging has notbeen completed (e.g., because wound 2700 continues to extend in one ormore directions), indication 2914 displayed by communications device 115may notify the user to move the device in an additional desireddirection (e.g., similar to indication 2910 to move communicationsdevice 115 in desired direction 2902). Once imaging has been completed,communications device 115 or another system communicatively coupled tocommunications device 115 may use the collected image and/or motion datato generate a three-dimensional model of wound 2700. Additionally,communications device 115 may be configured to generate and/or display arating of the user, based on the actual movement of communicationsdevice 115 matching or not matching the indicated desired direction 2902(and/or the actual positions matching or not matching the positionsindicated by crosshairs 2806 and/or boundary 2808).

Embodiments consistent with the present disclosure may include providingguidance to improve illumination conditions. For example, someembodiments may include detecting, based on an analysis of at least oneframe of the plurality of frames, that illumination conditions are notsatisfactory and, in response, displaying an indication on the mobiledevice to take an action to improve the illumination conditions. Asdiscussed previously herein, local illumination effects may result fromthe type of light source used to light the object, the distance of theobject from the light source, a viewing angle of the object, position ofthe object, ambient light conditions, flash usage, exposure time,shadows, and so forth. For example, in some embodiments, at least oneprocessor may be configured to derive at least one brightness and/orcontrast value that fails to meet a predetermined threshold to meetsatisfactory illumination conditions. Additionally or alternatively, insome embodiments, it may be determined that illumination conditions arenot met if convolution values cannot be calculated with a predeterminedthreshold value of certainty (e.g., 90%, 95%, etc.) based on a givenimage or plurality of images. By way of example, as illustrated in FIG.30, illumination conditions may not be satisfactory based on detectingthe presence of a shadow (e.g., shadow 3020) in one or more of frames inthe plurality of frames and/or detecting that the shadow is cast over awound (e.g., wound 3002) in the plurality of frames. Some embodimentsmay include: detecting, based on an analysis of at least one frame ofthe plurality of frames, the presence of a shadow in the plurality offrames; detecting that the shadow is cast over the wound in theplurality of frames; and determining, based on an analysis of the shadowin the plurality of frames, information related to an object casting theshadow. In some examples, a machine learning model may be trained usingtraining examples to detect presence of shadows in images and/or videos.An example of such training example may include a sample image and/or asample video, together with a label indicating whether the sample imageand/or the sample video includes a shadow. At least one frame of theplurality of frames may be analyzed using the trained machine learningmodel to detect the presence of shadow in the plurality of frames. Inother examples, histogram of at least a portion of the at least oneframe of the plurality of frames may be analyzed, for example bycomparing values of the histogram with thresholds, to detect thepresence of shadow in the plurality of frames.

Some embodiments of the present disclosure may include determining,based on an analysis of the shadow in the plurality of frames,information related to an object casting the shadow. The informationrelated to the object casting the shadow may, in some embodiments,include an identification of the object casting the shadow such as themobile device, a hand holding the mobile device, or another object ofmedical significance (e.g., a dipstick, testing kit, or any other typeof medical equipment). In other examples, the information related to theobject casting the shadow may include at least one of a type of theobject, a size of the object, a position of the object, or a shape ofthe object. Determining the information may include performing ananalysis on the detected shadow to determining physical parameters ofthe shadow (e.g., size, distance, angle, shape, etc.) and correlatingthe physical parameters with pre-stored information associated with thephysical parameters to identify the information related to the objectcasting the shadow. In some examples, a machine learning model may betrained using training examples to determine information related toobjects casting shadows from images and/or videos of the shadows. Anexample of such training example may include a sample image of a sampleshadow, together with a label indicating information related to anobject casting the sample shadow. The trained machine learning model maybe used to analyze shadow in the plurality of frames to determine theinformation related to the object casting the shadow. By way of example,determining the information may include calculating a convolution ofshadow 3020 in the image captured by communications device 115 in FIG.30 and deriving a result value of the calculated convolution indicativeof the size, shape, and/or distance of shadow 3020. Correlating theresult value to pre-stored information may including accessing at leastone data structure (e.g., database 146) storing physical parametersassociated with a plurality of objects (e.g., communications device 115,a hand, etc.). The result value may be compared with the physicalparameters in the at least one data structure to determine that shadow3020 is being cast by communications device 115.

Some disclosed embodiments may include determining a particular actionbased on the information associated with the object casting the shadowinformation and causing a performance of the particular action when theshadow is cast over the wound. The particular action may include anyaction that may be executed automatically by the mobile device (e.g.,modifying at least one parameter associated with the at least one imagesensor) or manually by the user (e.g., by moving the mobile device to adifferent location, moving the object casting the shadow so that it nolonger casts a shadow on the wound, interacting with other elements ofthe environment, etc.) to directly or indirectly cause theunsatisfactory illumination conditions to improve and/or becomesatisfactory. The at least one parameter may include image resolution,frame rate, gain, ISO speed, stereo base, lens, focus, zoom, colorcorrection profile, etc. associated with the image sensor (e.g., imagesensor 226) of the mobile device. In some embodiments, if theillumination conditions are unsatisfactory due to low brightness, theparticular action may include activating a flash feature associated withthe mobile device or turning on one or more other lights in theparticular room or environment. Causing a performance of the particularaction may include automatically triggering the mobile device to take anaction or by providing an indication to prompt the user to take aparticular action.

By way of example, as discussed above with reference to FIG. 30, someembodiments may include detecting the presence of shadow 3020 being castover wound 3002 and determining that the object casting the shadow ismobile device 115 or a hand holding mobile device 115. Based on thisinformation, the performance one or more particular actions may becaused to thereby improve the unsatisfactory illumination conditionsassociated with shadow 3020. For example, the zoom, focus, or lensassociated with image sensor 226 of mobile device 115 may be changed toimprove contrast and/or brightness. Additionally or alternatively,mobile device 115 may display an indication prompting the user to movethe object casting the shadow in one or more directions (e.g., bydisplaying indication 3010 prompting the user to “move the device closerto the wound” or the like) such that the object casting the shadow(e.g., mobile device 115) no longer casts a shadow over wound 3002.

FIG. 31 provides a flowchart of an example process 3100 for providingwound capturing guidance including steps 3102 through 3114. Steps 3102through 3114 may be executed by at least one processor (e.g., processingdevice 202 of server 145 or communications device 115), consistent withsome embodiments of the present disclosure.

Process 3100 may begin with step 3102. At step 3102, the at least oneprocessor may receive a plurality of frames from at least one imagesensor (e.g., image sensor 226) associated with a mobile device (e.g.,communications device 115), at least one of the plurality of framescontaining an image of a wound (e.g., wound 2700), consistent with someembodiments of the present disclosure. In other examples, receiving theplurality of frames by step 3102 may include at least one of reading theplurality of frames from memory, receiving the plurality of frames froman external device (for example, using a digital communication device),capturing the plurality of frames using the at least one image sensor,or generating the plurality of frames (for example, using a generativemodel).

Once the plurality of frames has been received, process 3100 may proceedto step 3104. At step 3104, the at least one processor may display areal time video including at least a portion of the plurality of framesand a visual overlay indicating a desired position of the wound on adisplay component of the mobile device (e.g., touch screen 218),consistent with some embodiments of the present disclosure. In someembodiments, the indication may alternatively or additionally beprovided through audible or other means (e.g., with speaker 222).

At step 3106, the at least one processor may detect that the wound is inthe desired position based on at least part of the plurality of frames,consistent with some embodiments of the present disclosure. For example,the at least one processor may detect that wound 2802A is in the desiredlocation due to its center 2804 coinciding with crosshairs 2806 in thecaptured images or due to wound 2802B coinciding with bounded shape2808, as illustrated in FIGS. 28A and 28B, respectively.

When the wound is in the desired position, process 3100 may proceed tostep 3108. At step 3108, the at least one processor may provide anindication to move the mobile device in a desired direction, consistentwith some embodiments of the present disclosure. For example, in FIG.29, the at least one processor may cause communications device 115 todisplay indication 2910 prompting the user to move the device in thedesired direction indicated by arrow 2902.

At step 3110, the at least one processor may receive motion data from atleast one motion sensor (e.g., motion sensor 228) associated with themobile device, consistent with some embodiments of the presentdisclosure. At step 3112, the at least one processor may detect that themobile device has moved in the desired direction, consistent with someembodiments of the present disclosure. For example, in FIG. 29, the atleast one processor may detect that communications device 115 has movedfrom the location denoted 115(1) to the location denoted 115(3) bymoving in the desired direction initially indicated by arrow 2902.Alternatively, the at least one may detect that communications device115 has moved in a direction that is different than the desireddirection (e.g., by moving to the location denoted 115(2)).

When the mobile device has moved in the desired direction, process 3100may proceed to step 3114. At step 3114, the at least one processor mayprovide an additional indication, consistent with some embodiments ofthe present disclosure. For example, in FIG. 29, the at least oneprocessor may cause mobile communications device 115 to displayindication 2914 once communications device has moved in the desireddirection to the location denoted by 115(3). Alternatively, the at leastone processor may cause communications device 115 to display indication2912 if communications device 115 moves in an incorrect direction to thelocation denoted by 115(2).

Embodiments consistent with the present disclosure provide systems,methods, devices, and computer readable media storing instructions forselective reaction to a failure to successfully complete a medicalaction using a medical image capturing application. As used herein, amedical image capturing application may include an applicationprogrammed into a user device (e.g., a computer, smartphone, tablet,etc.) configured to capture one or more images during medical testing,evaluation, and/or treatment. In some embodiments, a medical imagecapturing application may be configured to perform an analysis on theone or more captured images. The medical image capturing applicationmay, in some embodiments display or otherwise provide a user interfaceon the device, the user interface being configured to guide a patientthrough one or more steps for performing a medical action. As usedherein, a medical action may include any action in association with themedical testing, evaluation, and/or treatment of an individual patientand may be completed or attempted by a medical professional, the patientthemselves, or by any other caregiver. By way of example, FIG. 32illustrates a mobile communications device 115 with a medical imagecapturing application programmed thereon, consistent with some disclosedembodiments. The user interface of device 115 may allow a user tocapture images in a medical setting, for example, images of medicalsample 3200. Moreover, the image capturing application may guide theuser through a series of steps, for example by displaying indication3210 guiding the user to place a calibration element next to the medicalsample, by displaying other visual indications, by providing audibleguidance to the user, and so forth.

By way of further example, FIG. 33 provides a flowchart of an exampleprocess 3300 for selective reaction to a failure to successfullycomplete a medical action using a medical image capturing applicationincluding steps 3302 through 3328, consistent with some embodiments ofthe present disclosure. Steps 3302 through 3328 may be executed by atleast one processor (e.g., processing device 202 of communicationsdevice 115 or server 145). Process 3300 may begin at step 3302. At step3302, the at least one processor may provide a user interface forguiding a patient through steps for completing a medical action, asdiscussed above.

Consistent with disclosed embodiments, the plurality of steps mayinclude using at least one item of a medical kit. The at least one itemof a medical kit may include any medical items, such as disposable itemsused for treatment (e.g., bandages, gauze, tape, splints, salinesolution, etc.), medications (pain reliever, antibiotics, ointments,etc.), medical testing equipment (e.g., sample containers, dipsticks,etc.), other medical equipment (e.g., scissors, sutures, tweezers, coldcompresses, slings, etc.), or any other item associated with theprovision of healthcare. For example, in some embodiments, the at leastone item of the medical kit may include at least one of a dipstick(e.g., dipstick 450 depicted in FIG. 4B) and/or a calibrator (e.g.,colorized surface 132 depicted in FIG. 1A). The plurality of steps maybe completed by a medical professional, the patient, or any othercaregiver. As used herein, a step may refer to any portion of a medicalaction provided as a direction to a user, or it may constitute theentire medical action. In some disclosed embodiments, for example, usingat least one item of the medical test kit may include positioning acalibrator sticker, positioning a dipstick adjacent to a calibrator;dipping a dipstick in a medical sample, and/or blotting a dipstick. Byway of example, in FIG. 32, the particular medical action may be acertain test of medical sample 3200, and one of the plurality of stepsfor completing the test may include placing a calibrator (e.g.,colorized surface 132 in FIGS. 1A) next to medical sample 3200, asprovided in indication 3210 displayed on mobile communications device115.

Consistent with disclosed embodiments, the plurality of steps forcompleting the medical action may include capturing at least one imageof at least part of the at least one item of the medical kit using atleast one image sensor associated with a mobile device. For example, theat least one step may include taking a photo or video of an item ofmedical test kit and/or a wound to be treated with the at least one itemeither before, during, or after completion of another step (e.g.,applying a bandage, positioning a calibrator sticker or dipstick,dipping a dipstick in a medical sample, etc.). The at least one imagemay be captured, for example, with image sensor 226 of communicationsdevice 115, as illustrated in FIG. 2, and may be displayed on touchscreen 218 and/or processed by processing device 202, as discussedpreviously herein.

Embodiments consistent with the present disclosure may include detectinga failure to successfully complete the medical action. In someembodiments, detecting the failure may be based on an analysis of the atleast one captured image. For example, detecting the failure may includeperforming image processing on at least one captured image to determinewhether the at least one image is consistent with a successful use ofthe at least one item of the medical kit, or whether the at least oneimage was captured correctly. In some embodiments, a convolution of apart of the at least one captured image to derive a result value may becalculated, as previously discussed herein, and the result value may beused to detect the failure. Using the result value to detect a failureto use at least one item of a medical kit may include, for example,comparing the calculated result value to a threshold (for example, to athreshold based on a sample image of the at least one item that was usedproperly and/or based on a sample image of the at least one item thatwas used improperly). In one example, in response to a first resultvalue of the calculated convolution, a failure may be detected, and inresponse to a second result value of the calculated convolution, adetection of the failure may be avoided. In some examples, a machinelearning model may be trained using training examples to detect failuresto successfully complete medical actions from images and/or videos. Anexample of such training example may include a sample image, togetherwith a label indicating whether the sample image corresponds to afailure to successfully complete a medical action. In one example, thetrained machine learning model may be used to analyze the at least onecaptured image and detect the failure. A detection of a failure mayoccur, for example, during step 3304 of process 3300 in FIG. 33.Although step 3304 is illustrated as being subsequent to step 3302, afailure to complete a medical action may be detected during step 3302.For example, a failure may be detected before all of the plurality ofsteps are completed, or it may be detected after all of the plurality ofsteps are completed. If a failure is not detected, and thus the medicalaction is successfully completed, process 3300 may end at step 3306.Otherwise, if a failure is detected, process 3300 may proceed to step3310 (including steps 3312 through 3316), which involves the selectionof a reaction to the detected failure that is likely to bring asuccessful completion of the medical action, as discussed in furtherdetail herein.

In some embodiments, for example, a machine learning model may betrained using training examples to determine whether the at least oneitem is in a position consistent with proper use relative to otherreference positions in the at least one image frame. Examples of suchtraining examples may include sample images of correctly used sampleitems of a medical kit (e.g., a properly applied bandage, a properlypositioned dipstick and/or calibrator sticker, a medical samplecontainer provided with a proper sample, etc.) and sample images ofincorrectly used sample items (e.g., an improperly applied bandage, adipstick and/or calibrator stick placed upside-down with respect to theproper position, empty or partially filled medical sample containers,etc., medical sample containers with a wrong type of sample, etc.). Insome embodiments, training examples may also include sample images takenwith correct parameters (e.g., high resolution, proper lighting, etc.)and sample images taken with incorrect parameters (e.g., low brightness,low resolution, etc.). The trained machine learning model may be used toanalyze the at least part of the plurality of frames to detect that theuser correctly used, or incorrectly used, the at least one item of themedical kit as instructed, or to detect whether the user correctlycaptured the at least one image as instructed. In other examples, anobject detection algorithm may be used to analyze the at least part ofthe plurality of frames to determine an actual position of the at leastone item in the at least part of the plurality of frames, and the actualposition may be compared with a correct position of the at least oneitem to determine whether it was or was not properly used.

In some embodiments, detecting a failure may include identifying theparticular failure that occurred. For example, detecting a failure mayinclude at least one of detecting that the calibrator sticker isincorrectly positioned, that the dipstick is incorrectly positionedadjacent to the calibrator, that the dipstick is improperly dipped inthe medical sample, and/or that the dipstick is improperly blotted. Byway of example, as illustrated in FIG. 32, communications device 115 maydisplay an indication 3210 to place a calibrator (e.g., colorizedsurface 132 in FIG. 1A) next to medical sample 3200. Additionally oralternatively, indication 3210 may include a prompt to blot a dipstick(e.g., dipstick 450 in FIG. 4B) in medical sample 3200 and/or to placethe blotted dipstick next to the calibrator element. Subsequently,communications device 115 may provide guidance for an additional step tocapture at least one image of the blotted dipstick next to thecalibrator element. At least one processor (e.g., processing device 202of communications device 115 or server 145 in FIG. 2) may perform imageprocessing to detect a failure by determining that the dipstick wasimproperly dipped in the medical sample, for example due to a colorationof the dipstick being inconsistent with the coloration of a properlydipped dipstick. In some examples, an image classification algorithm maybe used to analyze the at least one captured image and determine a typeof the failure. For example, each class may correspond to a differenttype of failure, and the type of failure may be determined based on theclass assigned to the at least one captured image by the imageclassification algorithm.

In some embodiments, detecting a failure may be based on a timingassociated with a detected action performed by the user. For example, insome embodiments, the detected failure may include a failure to capturethe at least one image within a particular time window, or it mayinclude detecting that the user interface was shut down beforecompleting at least one of the steps for performing a medical action.The particular time window may be based on a time of a physical actioninvolving the at least one item (e.g., applying a bandage, collecting amedical sample, blotting a dipstick, etc.) and/or a time of user actionin the user interface (e.g., an interaction with touch screen 216 ofcommunications device 115 in FIG. 2, thereby confirming completion of atleast one of the plurality of steps, causing at least one image to becaptured, or shutting down the provided user interface). By way ofexample, it may only be possible for a valid result to be determinedfrom a dipstick (e.g., dipstick 450 in FIG. 4B) within a window of 15 to30 minutes after the dipstick has been blotted into medical sample 3200.Thus, at least one processor associated with communications device 115(e.g., processor 202) may detect a failure to properly blot the dipstickif the user captures at least one image outside the 15 to 30 minutewindow.

Embodiments consistent with the present disclosure may includeselecting, from one or more alternative reactions, a reaction to thedetected failure likely to bring a successful completion of the medicalaction. Additionally, some disclosed embodiments may include providinginstructions associated with the selected reaction. A reaction asreferred to herein may include a medical action that is made in responseto a failure to bring completion of the medical option that may remedythe error caused by the respective failure. For example, in someembodiments, alternative reactions may include triggering a provision ofan additional medical kit to the patient, triggering an approach to thepatient by a person, or triggering a provision of additional guidance tothe patient using the user interface. Triggering a particular action asused herein may refer to causing a mobile device to automaticallyexecute the action (e.g., automatically adjusting parameters of imagesensor 226 of communications device 115), providing instructions on theparticular mobile device prompting the user to perform the particularaction, or by providing instructions through one or more externaldevices (e.g., in FIG. 1A, communications device 125 associated withmedical practitioner 120, server 145 associated with medical analysisunit 140, communications device 165 associated with healthcare provider160, communications device 175 associated with insurance company 170,etc.) to prompt the associated user to perform the particular action.For example, the provided instructions may be configured to cause theprovision of an additional medical kit to the patient by another person(e.g., a medical professional or other caregiver such as medicalpractitioner 120) or to alert the person to approach the patient.

The one or more alternative reactions may be stored in at least one datastructure, such as a database (e.g., database 146). In some embodiments,the plurality of alternative reactions may be mapped to one or more of avariety of different variables associated with the particular medicalaction and/or patient, and the selection of the alternative reaction maybe based on the particular variables associated with the particularmedical action and/or patient. For example, the selection of thereaction may be based on a type of the failure detected, a result of thedetected failure, the particular step for performing a medical actionfailed, a characteristic of the patient, and/or other factors that mayaffect the appropriate response to a detected failure. In someembodiments, detecting a failure may include identifying the one or morefailed steps for performing a medical action and selecting a reaction isbased on the one or more failed steps identified.

A type of failure may refer to a categorization of the failure, such aswhether the failed action was the use of at least one item of a medicalkit or a failure to capture at least one image with the mobile device asinstructed. Each different particular failure may have different resultsthat may influence which alternative reaction should be selected. Forexample, a failure to capture an image due to poor lighting may not havenegative results and may require a simple reaction (e.g., turn on aflash component of communications device 115). In such a situation, itmay be most appropriate to simply guide the patient to perform thesimple reaction than to trigger additional assistance by a medicalprofessional. However, if the detected failure is one that is likely tocause an emergency (e.g., a failure that may result in substantialinjury), then it may be more appropriate to have a medical professional(e.g., medical practitioner 120 in FIG. 1A) or other caregiver provideassistance to the patient. Characteristics of a patient may be stored inat least one data structure (e.g., database 146), and may include anydemographic information, medical information, or any other factorsassociated with the patient that may increase or decrease the likelihoodthat they are able to follow provided instructions and/or perform aparticular action. Some example characteristics may also include factorsthat affect the risk that the detected failure will result in furtherinjury or complications. Some example characteristics may include age,sex, education, lifestyle factors, location (e.g., whether the patientis at home or at a certain type of hospital), whether the patient ishandicapped, preexisting conditions, and the like.

A characteristic may refer to a single one of these characteristics, orit may refer to a given combination of characteristics associated withthe patient. A reaction based on a characteristic of a patient may bebased on historic statistical data that may be used to determine acertainty that a patient has a particular characteristic, and acharacteristic may be attributed to a patient if the certainty levelexceeds a certain threshold (e.g., 90%, 95%, etc.). The givencharacteristic or characteristics of the patient may also be used todetermine an urgency level. In some embodiments, the determined urgencylevel may be used to schedule and prioritize medical actions betweenhigh urgency/risk patients and low urgency/risk patients. For example,if there is a lack of availability of medical professionals or othercaretakers, a selected reaction to a failure resulting in an emergencywill be given higher priority than if the selected reaction has lowurgency.

In some embodiments, the selected reaction may depend on whether thefailure necessitates a usage of an alternative item to the at least oneitem of the medical kit for a successful completion of the medicalaction. Moreover, the selected reaction may depend on whether thefailure necessitates a usage of an alternative item that is not in themedical kit (e.g., because it has already been used). In one example, itmay be determined that the failure necessitates the usage of thealternative item in response to a failure to capture the at least oneimage within a particular time window (for example, a failure to capturean image of a dipstick in a particular time window after dipping it in asample, after receiving an instruction to dip it in the sample, afteracknowledging that it was dipped in a sample, etc.). In another example,it may be determined that the failure necessitates the usage of thealternative item based on an analysis of the at least one image. Forexample, a convolution of a part of the at least one captured image maybe calculated to derive a result value. In response to a first resultvalue, it may be determined that the failure necessitates the usage ofthe alternative item, and in response to a second result value, it maybe determined that the failure does not necessitate the usage of thealternative item. In another example, the at least one captured imagemay be analyzed to determine whether the at least one item iscontaminated, and the necessitated usage of the alternative item may bedetermined in response to the determined contamination.

In some embodiments, the selected alternative reaction may be based on acombination of the factors discussed above. For example, as discussedabove, embodiments consistent with the present disclosure may includedetermining that the failure necessitates a usage of an alternative itemto the at least one item of the medical kit for a successful completionof the medical action. In one example, when the medical kit includes thealternative item, the selected reaction may include at least one oftriggering an approach to the patient by a person or triggering aprovision of additional guidance to the patient using the userinterface. In another example, when the medical kit does not include thealternative item, the selected reaction may include at least one oftriggering a provision of an additional medical kit to the patient ortriggering a performance of the medical action by a medicalprofessional. In some examples, determining that the failurenecessitates the usage of the alternative item to the at least one itemof the medical kit for the successful completion of the medical actionmay be based on the type of the failure. In one example, the type of thefailure may be determined as described above. In one example, inresponse to a first type of failure, it may be determined that thefailure necessitates the usage of the alternative item, and in responseto a second type of failure, it may be determined that the failure doesnot necessitate the usage of the alternative item. In some examples,determining that the failure necessitates the usage of the alternativeitem to the at least one item of the medical kit for the successfulcompletion of the medical action may be based on a property of a usageof the user interface. One example of such property may include a usageof a particular functionality of the user interface. Another example ofsuch property may include a time duration associated with completion ofan action using the user interface. In one example, the type of thefailure may be determined as described above. In one example, inresponse to a first property of the usage of the user interface, it maybe determined that the failure necessitates the usage of the alternativeitem, and in response to a second property of the usage of the userinterface, it may be determined that the failure does not necessitatethe usage of the alternative item.

By way of example, FIG. 33 provides a flowchart of an example process3300 for selective reaction to a failure to successfully complete amedical action using a medical image capturing application, consistentwith some embodiments of the present disclosure. If a failure isdetected, as discussed previously, process 3300 may proceed to step 3310(including steps 3312 through 3316), which involves selecting one of theone or more alternative reactions based on the availability of anecessary alternative item (e.g., step 3312) and patient characteristics(steps 3314 and 3316). For example, at step 3312, at least one processor(e.g., processor 202 of communications device 115 or server 145) maydetermine whether the particular detected failure necessitates the useof an alternative item in another medical kit. If the detected failurenecessitates the use of an additional or alternative item in anothermedical kit that is not in the present medical kit, process 3300 mayproceed to step 3314. For example, process 3300 may proceed to step 3314if the selected reaction requires an additional dipstick or bandagebecause the dipstick or bandage available in the medical kit has alreadybeen used. If the detected failure does not necessitate the use of anadditional item in another medical kit, then process 3300 may proceed tostep 3316. For example, process 3300 may proceed to step 3316 if theadditional item needed is already available in the present medical kit,or if an additional item is not required for the selected alternativereaction whatsoever.

As discussed above, embodiments consistent with the present disclosuremay include determining that the failure necessitates a usage of analternative item to the at least one item of the medical kit for asuccessful completion of the medical action. In one example, if thefailure necessitates a usage of an alternative item and the patient hasa first characteristic, a provision of an additional medical kit may betriggered. For example, the first characteristic may be indicative thatthe respective medical action will likely be completed upon providingthe patient with an additional medical kit (e.g., because the patient islikely to complete the medical action). Accordingly, the additionalmedical kit will be provided for self-administration of the medicalaction by the patient. In another example, if the failure necessitates ausage of an alternative item and the patient has a secondcharacteristic, the performance of the medical action by a medicalprofessional may be triggered. For example, the second characteristicmay be indicative that the respective medical action will not likely becompleted upon providing the patient with an additional medical kit(e.g., because the patient is unlikely to be able to complete themedical action on their own). Accordingly, the medical action will becompleted by a medical professional instead. In some examples, thecharacteristic of the patient may be determined by accessing a databaseincluding characteristics of patients. In another example, thecharacteristic of the patient may be read from memory, may be receivedfrom an external device, may be received from a user of the userinterface (for example, from the patient, from a caregiver of thepatient, etc.), and so forth. In yet another example, the characteristicof the patient may be determined based on an analysis of the interactionof the patient with the user interface. For example, a classificationalgorithm may be used to analyze the interaction and to classify thepatient to different categories of patients.

By way of example, at step 3314 of process 3300, at least one processor(e.g., processor 202 of communications device 115 or server 145) maydetermine whether the patient has a particular characteristic indicativeof the patient's inability to complete the particular medical action andproceed to step 3322. At step 3322, the at least one processor maytrigger a performance of the medical action by a medical professional(e.g., medical practitioner 120) instead of the patient. Alternatively,if the patient does not have the particular characteristic or hasanother characteristic indicative of the patient's ability to completethe particular action, the at least one processor may proceed instead tostep 3324, where it may trigger the provision of an additional medicalkit to the patient.

As discussed above, some disclosed embodiments may include determiningthat the failure does not necessitate a usage of an alternative item tothe at least one item of the medical kit for a successful completion ofthe medical action. In one example, if the failure does not necessitatethe usage of an alternative item and the patient has a firstcharacteristic, an approach to the patient by a person may be triggered.For example, the first characteristic may be indicative that therespective medical action will not likely be completed upon providingthe patient with an additional medical kit (e.g., because the patient isunlikely to be able to complete the medical action on their own).Accordingly, the patient will be approached by another individual tohelp complete the medical action. In another example, if the failuredoes not necessitate the usage of an alternative item and the patienthas a second characteristic, the provision of additional guidance to thepatient using the user interface may be triggered. For example, thesecond characteristic may be indicative that the user will likelycomplete the medical action upon further instruction (e.g., because thepatient is capable of following the additional instructions).

By way of example, at step 3316 of process 3300, at least one processor(e.g., processor 202 of communications device 115 or server 145) maydetermine whether the patient has a particular characteristic indicativeof the patient's inability to complete the particular medical action. Ifthe patient has the particular characteristic, process 3300 may proceedto step 3326. At step 3326, the at least one processor may trigger anapproach to the patient by another person (e.g., medical practitioner120 or another caregiver) to help the patient complete the medicalaction. Alternatively, if the patient does not have a firstcharacteristic or has a second characteristic indicative of a capabilityto follow additional instructions, process 3300 may instead proceed tostep 3328. At step 3328, the at least one process may trigger theprovision of additional guidance to the patient through a user interfaceof device 115 (e.g., touch screen 218).

Embodiments consistent with the present disclosure may include systems,methods, devices, and computer-readable media storing instructions fordisplaying an overlay on wounds. As referred to herein, a wound mayinclude any injury to the human body. For example, wounds may be openwounds resulting from penetration (e.g., puncture wounds, surgicalwounds and incisions, thermal, chemical, or electric burns, bites andstings, gunshot wounds, etc.) and/or blunt trauma (e.g., abrasions,lacerations, skin tears), or they may include closed wounds (e.g.,contusions, blisters, seromas, hematomas, crush injuries, etc.). Somenon-limiting examples of a wound may include a chronic wound, acutewounds, ulcer (such as venous ulcer, arterial ulcer, diabetic ulcer,pressure ulcer, etc.), infectious wound, ischemic wound, surgical wound,radiation poisoning wound, and so forth.

An overlay as used herein may include one or more elements of a userinterface that are superimposed on an image, a video, or on theenvironment. The overlay may be superimposed on the user interface, forexample, by displaying an image on a user interface along with theoverlay, or it may include displaying the overlay on the user interfacesuch that it appears to be superimposed on the image as viewed directlyby the user (e.g., augmented reality glasses). That is, an image as usedherein may refer to an image that is displayed on a user interface, orit may merely refer to the manifestation of the visual perception of asubject. For example, an image may include an image of a subjectdisplayed on a user interface, an image that is captured by an imagingdevice, or an image as viewed by the human eye. As used herein, anoverlay or portions thereof may also be referred to as an indication,for example as an indication of a condition of a wound, that issuperimposed on an image. However, in some embodiments, an overlay mayalso include one or more non-superimposed indications or elements inaddition to one or more superimposed elements.

By way of example, FIG. 34 illustrates an example mobile communicationsdevice 115 that is configured to display overlay 3420 on an image 3410of wound 3400 in a video feed displayed on touch screen 218, consistentwith some embodiments of the present disclosure. Overlay 3420, asdisplayed on touch screen 218, may include elements 3422 and 3424 thatare superimposed on wound 3400 in image 3410. Although not illustratedherein, the image as displayed on touch screen 218 may be arepresentation of the image viewed by a user through a transparentsurface (e.g., a window or augmented reality glasses) that is configuredto display overlay 3420 such that elements 3422 and 3424 aresuperimposed on wound 3400 to the user. Accordingly, unless expresslystated otherwise, touch screen 218 is to be understood to illustratethat the disclosed overlays may be displayed on any graphical userinterface, such as a non-touch screen display or a transparent surfaceconfigured for displaying an overlay.

Embodiments consistent with the present disclosure may include receivinga real time video feed. A real time video feed as referred to herein mayinclude a plurality of images that have been captured by an imagingdevice, or it may include a plurality of images that are being capturedin real-time. The subject of the captured images may include a singlewound or a plurality of wounds. Some embodiments consistent with thepresent disclosure may also include selecting a wound from a pluralityof wounds in the image. The selected wound may, for example, be a woundthat a user selects to be provided with an overlay or element of anoverlay. A wound may be selected through any suitable means according tothe type of interface used. For example, selecting a wound may includeclicking on a displayed image of a wound, touching a displayed imagewound on a touch screen, providing verbal or non-verbal (e.g., gestures)instructions to the user interface to select the wound, and the like. Byway of example, FIG. 35 illustrates an example mobile communicationsdevice 115 that is configured to display overlay 3520 on an image ofwound 3500 in a video feed displayed on touch screen 218, consistentwith some embodiments of the present disclosure. The video feed on whichoverlay 3520 is displayed may be previously recorded or viewed and/ordisplayed in real-time and may include images of both wound 3500 andwound 3502. Although overlay 3520 is displayed on touch screen 218 asbeing superimposed on wound 3500, touch screen 218 may alternatively oradditionally display an overlay on wound 3502 if wound 3502 is selectedby the user. In some examples, at least part of the video feed may beanalyzed to automatically select the wound from the plurality of wounds.For example, information related to a particular wound may be used togenerate the overlay, for example by visually presenting at least partof the information related to the particular wound in the overlay. Theinformation related to the particular wound may be based on image-basedinformation associated with at least one previously captured image ofthe particular wound, for example as described below. Further, theinformation related to the particular wound may be used to select thewound corresponding to the information from the plurality of wounds. Inone example, the information related to the particular wound may includea size of the particular wound, and a wound of the plurality of woundswith a size nearest to the size of the particular wound may be selected.The size of each wound of the plurality of wounds may be determined byanalyzing the at least part of the video feed. In another example, theinformation related to the particular wound may include a shape of theparticular wound, and a wound of the plurality of wounds with a shapemost resembling to the shape of the particular wound may be selected.The shape of each wound of the plurality of wounds may be determined byanalyzing the at least part of the video feed. In some examples, amachine learning model may be trained using training examples to selecta wound of a plurality of wounds in an image that corresponds toparticular wound characteristics. An example of such training examplemay include a sample image depicting a sample plurality of wounds andone or more sample characteristics of a desired wound, together with alabel indicating a wound of the sample plurality of wounds correspondingto the one or more sample characteristics. The trained machine learningmodel may be used to analyze the at least part of the video feed andautomatically select the wound from the plurality of wounds.

Embodiments consistent with the present disclosure may include receivingimage-based information associated with at least one previously capturedimage of a wound. Image-based information as referred to herein mayinclude data comprising the previously captured image (e.g., digitalimages formatted with PNG, JPEG, GIF, and the like), data associatedwith the previously captured image (e.g., a time of capture of theimage, patient information, wound information, etc.), or any datagenerated based on the image using computer vision and/or imageprocessing as discussed herein. The previously captured image may becaptured at any time prior to a time at which the video feed is captured(e.g., seconds, hours, days, weeks, etc.), for example at least one daybefore the video feed is captured.

In some embodiments, receiving the image-based information may includeaccessing a plurality of records, selecting a record corresponding tothe wound of the plurality of records based on the video feed, andobtaining the image-based information from the selected record. A recordas used herein may refer to a digital file or collection of data orinformation that has been previously recorded or saved into storage.Records may be stored in any suitable data structure, such as a database(e.g., databases 146 and 166 in FIG. 1A) or local memory (e.g., memorydevice 234 in FIG. 2). In some embodiments, each record in the pluralityof records may correspond to a different wound. In some examples, atleast part of the video feed may be analyzed to determinecharacteristics of the wound (such as size, shape, location on the body,tissue composition, etc.), and the record corresponding to the wound maybe selected from the plurality of records based on the determinedcharacteristics. For example, a record that best matches the determinedcharacteristics of the plurality of records may be selected. In someexamples, a machine learning model may be trained using trainingexamples to select records corresponding to wounds based on images ofthe wounds. An example of such a training example may include aplurality of sample records and a sample image of a sample wound,together with a label indicating a record from the plurality of samplerecords corresponding to the sample wound. The trained machine learningmodel may be used to analyze the at least part of the video feed andselect the record from the plurality of records. Some disclosedembodiments may include receiving second, third, or any furtherimage-based information associated with a second, third, or further atleast one previously captured image of the wound. In some examples, arecord may include one or more images and associated data of therespective wound. For example, a record corresponding to a particularwound may contain several previously captured images, and the previouslycaptured images may have been captured at different times. By way ofexample, in FIG. 34, superimposed elements 3422 and 3424 may correspondto images of wound 3400 that were captured at different points in timein the past. Elements 3422 and 3424 of overlay 3420 may be generated,for example as discussed with further detail herein, based on imagescontained in one or more records stored in a database (e.g., databases146 and 166 in FIG. 1A) or local memory (e.g., memory device 234 in FIG.2).

Embodiments consistent with the present disclosure may includegenerating an overlay using a video feed and image-based information.The overlay may include an indication of a condition of the wound in theat least one previously captured image. An indication as referred toherein may refer to a superimposed element or a non-superimposed elementof an overlay. In some non-limiting examples, an indication may betextual, graphical, a combination of a text with graphics, and so forth.In another non-limiting example, the information contained in theindication may be provided audibly. In some embodiments, for example,the overlay may include an indication of a capturing time associatedwith a first, second, third, and/or further at least one previouslycaptured image. The capturing time may be an absolute point in time(i.e., a specific date and time of capture) or a relative point in time(e.g., one week ago, two months ago, etc.) In some embodiments, theoverlay may include a second, third or further indication. The second,third, or further indication may be an indication of a condition of thewound in the second at least one previously captured image. Thecondition of the wound in the second at least one previously capturedimage may differ from the condition of the wound in the at least onepreviously captured image. In other words, in some embodiments, theoverlay may include indications corresponding to any number ofpreviously captured images to provide information associated with acondition of the wound in each respective previously captured image.

By way of example, in FIG. 35, the generated overlay 3520 may includenon-superimposed elements 3500A, 3522A, and 3524A, and superimposedelements 3522B and 3524B. Non-superimposed elements 3500A, 3522A, and3524A may correspond with wound 3500 and superimposed elements 3522A and3524B, respectively. Elements 3522A and 3522B may be generated using thevideo feed (e.g., to correspond a location of the wound with a locationof superimposed element 3522B) and image-based information from a recordassociated with at least one previously captured image of wound 3500.Non-superimposed element 3522A may include an indication of a relativecapturing time of the at least one image (i.e., “Past Condition: 2Weeks”) associated with the indication. Moreover, as illustrated in FIG.34, overlay 3420 may include multiple indications associated withpreviously captured images of wound 3400. For example, element 3422 maycorrespond to an image captured at a more recent point in time (e.g., aweek), whereas element 3424 may correspond to an image captured at aless recent point in time (e.g., a month). Although not explicitlyillustrated therein, superimposed elements 3422 and 3424 may include anindication of a capturing time of each respective image, or overlay 3420may include additional non-superimposed elements including an indicationof the capturing time of each respective image.

Consistent with disclosed embodiments, generating the overlay mayinvolve image processing, as discussed previously herein. In someembodiments, generating the overlay may include combining the geometryand attributes of multiple different data sets (e.g., image dataassociated with the video feed and image data associated with at leastone previously captured image). In some non-limiting examples,generating the overlay may include combining Vector data sets, Rasterdata sets, or both. Vector data, for example, may provide data forcorrelating features of an image with its geometry and attribute. Thegeometry data may include data indicative of points (0-dimensional),lines (1-dimensional), polygons (2-dimensional), and/or volumes(3-dimensional) associated with a certain feature. Some non-limitingexamples of Vector data formats include Shapefile, geodatabase featureclass, GML, KML, and GeoJSON. Raster data may include Raster grids,which are typically made up of square or rectangular grids with a singlevalue corresponding to each cell, thus representing a 2-dimensionalarray of samples. Some non-limiting overlaying functions may includeintersection (including only features present in all input layers),union (including only features occurring in either or both inputlayers), subtraction (excluding overlapping features of input layers),symmetric difference (including all features that occur in one of theinput layers but not all input layers), identity (for one of the inputlayers, merging features of overlapping input layers), cover (similar tounion, where one layer is retained in areas of overlapping features),and clip (cropping an input layer to areas where features of other inputlayers overlap).

The digital data for providing the overlay may be initially generatedwith photo interpretation of the images in the video feed and/or with atleast one previously captured image. Photo interpretation may beconfigured to capture 2-dimensional and 3-dimensional data, withelevations measured using one or more photogrammetric methods (e.g.,stereophotogrammetry). Although disclosed embodiments may be describedwith reference to previously captured images or images from a videofeed, other remote sensing technologies may be used to collect2-dimensional and 3-dimensional data associated with a wound or otherphysical feature to generate an overlay, consistent with someembodiments of the present disclosure. Some non-limiting examples ofremote sensing technologies include radar, LIDAR, radiometry, andmulti-spectral mapping.

Some embodiments may include a system for displaying an overlay on awound, the system comprising at least one processing unit configured to:receive a real time video feed; receive image-based informationassociated with at least one previously captured image of a wound;generate, using the video feed and the image-based information, anoverlay including an indication of a condition of the wound in the atleast one previously captured image; and display, on at least one userinterface, the overlay, wherein the at least one user interface isconfigured to display the overlay in a position associated with aposition of the wound in the video feed.

Other embodiments may include a method for displaying an overlay on awound in, the method comprising: receiving a real time video feed;receiving image-based information associated with at least onepreviously captured image of a wound; generating, using the video feedand the image-based information, an overlay including an indication of acondition of the wound in the at least one previously captured image;and displaying, on at least one user interface, the overlay, wherein theat least one user interface is configured to display the overlay in aposition associated with a position of the wound in the video feed.

In some disclosed embodiments, generating an overlay may includecalculating a convolution, as discussed previously herein, of at leastpart of the at least one previously captured image to derive a resultvalue. The result value may then be used to generate an indicationassociated with the result value. For example, in response to a firstresult value, the overlay may include a first indication of thecondition of the wound in the at least one previously captured image. Inresponse to a second result value, the overlay may include a secondindication of the condition of the wound in the at least one previouslycaptured image, where the second indication differs from the firstindication. By way of example, in FIG. 34, elements 3422 and 3424 maycorrespond to alternative indications (e.g., a first indication or asecond indication) of a shape of a wound in at least one previouslycaptured image. A result value of the calculated convolution of thewound in the at least one previously captured image may correspond tothe shape indicated by element 3422 and not element 3424. Accordingly,element 3422 may be displayed on overlay 3420 instead of element 3424.

Consistent with the present disclosure, a condition of a wound asreferred to herein may refer to a medical condition (e.g., infection) orany other physical parameter associated with a wound. For example, insome embodiments, an indication of a condition of a wound included in anoverlay may include at least one of an indication, including in somecases visual indications, of a contour or shape of a wound at least onemeasurement of the wound in the at least one previously captured image(e.g length, an area, a volume, or a depth of the wound, etc.), a tissuetype (e.g., granulation tissue, slough tissue, eschar, necrotic tissue,scab, hematoma, tendon, ligament, bone, infected tissue, non-infectedtissue, etc.) of at least one segment of the wound in the at least onepreviously captured image corresponding to a tissue type, a color of aportion of the wound in the at least one previously captured image, aseverity of the wound in the at least one previously captured image, orany other conceivable characteristic that may be associated with awound. By way of example, in FIG. 35, overlay 3520 may includenon-superimposed elements 3500A, 3522A, and 3524A, each of which providea textual indication of length, width, and depth measurements, color ofthe wound, and infection status of the wound. In contrast, superimposedelements 3522B and 3524B provide a visual indication of the size, shape,and contours of wound 3500.

Some embodiments of the present disclosure may include determining acondition of a wound in at least one image. A condition associated witha wound may, for example, be determined by a medical professional (e.g.,medical practitioner 120 in FIG. 1A) and placed into a recordcorresponding to the wound (e.g., saved in database 146). However, someembodiments of the disclosure may include using machine learning, aspreviously discussed herein, to determine a condition of a wound or toestimate and/or interpolate a condition of a wound. For example, in someembodiments, a machine learning model (e.g., a generative model, such asa generative adversarial network, a transformers-based model, etc.) maybe trained using training examples to determine one or more conditionsof a wound in one or more images or to estimate and/or interpolate acondition of a wound at a certain point in time. Examples of trainingexamples for determining a condition of a wound may include sampleimages of wounds having known conditions (e.g., an infected wound withpredetermined measurements, color, tissue types, etc.). For estimatingand/or interpolating the condition of a wound, examples of trainingexamples may include sets of sample images of wounds with knownconditions, where each set may include multiple images of the same woundat multiple times with time stamps for the times at which each of theimages is captured. The trained machine learning model may be used toanalyze 3D information associated with one or more images in the videofeed and/or at least one previously captured image to determine acondition of the wound in the one or more images or to estimate and/orinterpolate a condition of a wound at a certain point in time.

As previously discussed, some disclosed embodiments may includeextrapolating and/or determining a condition of a wound at a certainpoint in time. For example, in some disclosed embodiments, a conditionof a wound in the at least one previously captured image may correspondto a first point in time, and a condition of the wound in a second atleast one previously captured image may correspond to a second point intime. Where the condition of a wound in a different or third point intime is unknown, the image-based information associated with the firstat least one previously captured image and the second image-basedinformation may be used to determine a condition of the woundcorresponding to a third point in time. A third indication in theoverlay indicating the condition of the wound corresponding to the thirdpoint in time may be included in the overlay. In one example, the thirdpoint in time may be a future point in time, and the condition of thewound corresponding to the third point in time may be a predictedcondition of the wound in the future. In another example, the thirdpoint in time may be a point in time between the first point in time andthe second point in time, and the condition of the wound correspondingto the third point in time may be an interpolation. In yet anotherexample, the third point in time may be a point in time before the firstpoint in time and the second point in time, and the condition of thewound corresponding to the third point in time may be an extrapolation.In an additional example, the third point in time may be a point in timeafter the first point in time and the second point in time, and thecondition of the wound corresponding to the third point in time may bean extrapolation. In some examples, the first point in time may alsorefer to one or more times at which one or more images in the video feedare captured.

By way of example, in FIG. 34, element 3424 may correspond to a firstpoint in time at which an image of wound 3400 was previously captured,and image 3410 of wound 3400 may be captured at a second point in time.In some embodiments, at least one processor (e.g., processing device 202of mobile communications device 115 or server 145) may use machinelearning or other computerized methods to interpolate a condition ofwound 3400 at a third point in time between the first point in time andthe second point in time. An indication of the condition of wound 3400may be displayed, for example, as element 3422 in overlay 3420.

By way of further example, in FIG. 35, elements 3522A and 3522B maycorrespond to a first point in time at which an image of wound 3500 wascaptured, and the image of wound 3500 displayed on overlay 3520 may becaptured at a second point in time. In some embodiments, at least oneprocessor may use machine learning or other computerized methods (e.g.,processing device 202 of mobile communications device 115 or server 145)to predict a condition of wound 3500 at a third point in time (i.e., afuture point in time) that is after the first point in time and thesecond point in time. An indication of the condition of wound 3400 maybe displayed, for example, as elements 3524A and 3524B, including anindication of the future point in time (i.e., “Anticipated Condition: 2Weeks”). Similarly, in some embodiments, if the condition of wound 3500at the point of time associated with element 3524A and 3524B is known(e.g., indication 3524B instead represents an image of wound 3500captured at a first point in time, and the image of wound 3500 displayedon touch screen 218 illustrated in FIG. 35 instead represents apreviously captured image captured at a second point in time), the atleast one processor may extrapolate a condition of the wound 3500 at athird point in time that is after the first point in time and after thesecond point in time. In some embodiments, the at least one processormay extrapolate a condition of the wound 3500 at a third point in timethat is before the first point in time and the second point in time. Inthis example, indications 3522A and 3522B may correspond to theestimated condition of wound 3500 at the third point in time. In yetother embodiments, the at least one processor may interpolate acondition of the wound 3500 at a third point in time that is after thefirst point in time but before the second point in time.

Embodiments consistent with the present disclosure may includedisplaying the overlay on at least one user interface. The at least oneuser interface may be configured to display the overlay in a positionassociated with a position of the wound in the video feed. In someembodiments, the at least one user interface may be associated with amobile device. For example, the user interface may be displayed alongwith the video feed in mobile devices with integrated display elements,such as smartphones or tablets. However, as discussed previously herein,the at least one user interface is not to be limited to such mobiledevices. For example, the at least one user interface may be associatedwith an extended reality system. Some non-limiting examples of extendedreality systems include virtual reality systems, augmented realitysystems, mixed reality systems, augmented reality glasses, head mountedextended reality systems, wearable extended reality systems, and soforth.

For example, in some embodiments, the overlay may be displayed using atransparent optical system included in a wearable device, and the videofeed may be captured using an image sensor included in the wearabledevice. Accordingly, a wound may be visible to a user wearing thewearable device through the transparent optical system, and the displayof the overlay may be configured to make the overlay appear to the userwearing the wearable device at least partly over the wound. By way ofexample, the images displayed on touch screen 218 of mobilecommunications device 115 illustrated in FIGS. 34 and 35 may insteadrepresent an image of the wound as viewed by a user that is directlyviewing the wound through a transparent surface. In other words, theimage of wounds 3400 and 3500 may, in the case that the user interfaceis associated with a transparent optical system, represent actual imagesof the wounds instead of computer-generated images. In this example,overlays 3420 and 3520 may be displayed on the transparent surface, forexample using a projector or using a transparent display screen.

In some embodiments, the overlay may be displayed on the user interfacefeed in real time. That is, the overlay may be displayed on the userinterface virtually instantaneously as the video feed is captured.Accordingly, if the video feed is captured using a device associatedwith the user interface (i.e., if the video is captured from a similarpoint of view to the point of view of the user), the user interface maycause the overlay to appear as if the overlay is superimposed on theactual image (e.g., augmented reality systems). In some embodiments, thevideo feed may be captured at a separate location than the userinterface, and the user interface may be configured to use the videofeed to display a plurality of new computer-generated images includingthe overlay depicting the separate location, such that the userinterface may cause the separate location to appear as if it is in thesame location as the user (e.g., virtual reality systems).

In some embodiments consistent with the present disclosure, the at leastone user interface may be configured to automatically adjust theposition of the displayed overlay based on detected movement of thedevice capturing the video feed. For example, a user interface with adisplayed overlay may automatically move the overlay in response tomovement of the device, such that the displayed overlay appears to movein conjunction with the image of the wound. In some embodiments, themovement may be detected based on an analysis of the video feed. Forexample, the motion may be determined, at least in part, based on ananalysis of the plurality of images in the video feed captured by atleast one image sensor (e.g., image sensor 226 depicted in FIG. 2) ofthe device. The analysis of the plurality of images may be conductedwith an egomotion algorithm, for example. Additionally or alternatively,the movement may be detected based on information received from at leastone motion sensor associated with the mobile device. Motion sensors(e.g., motion sensor 228 depicted in FIG. 2) may include accelerometers,gyroscopes, or any other sensor configured to measure acceleration,gravity, speed of revolution, curl vector values, or drift of the mobiledevice. In some embodiments, a correspondence between the actualmovement of the device detected using the image sensors and/or motionsensors and a computed position of the overlay on the image may bedetermined (e.g., using a local and/or global positioning system) andused to modify the display of the overlay on the user interfaceaccordingly.

By way of example, in FIG. 34, mobile communications device 115 may bemoved by the user to different positions, and the position of overlay3420 on touch screen 218 may move in order to maintain its positionrelative to the displayed image of wound 3400. The movement may be basedon one or both of image data collected from at least one image sensor(e.g., image sensor 226 in FIG. 2) or at least one motion sensor (e.g.,motion sensor 228 in FIG. 2) associated with mobile communication device115. For example, if mobile communications device 115 moves upwardswithout changing orientation, overlay 3420 as displayed on touch screen218 will move downwards along with the corresponding portion of wound3400. In another example, if mobile communications device 115 were movedto view wound 3400 at an angle perpendicular to the radial portion ofthe forearm, the orientation of overlay 3420 as displayed on touchscreen 218 would change consistently with the changing orientation ofthe wound in the video feed.

FIG. 36 provides a flowchart of an example process 3600 for displayingan overlay on wounds in a video feed. Process 3600 includes steps 3602through 3608, which may be executed by at least one processor (e.g.,processing device 202 of communications device 115 or server 145),consistent with some embodiments of the present disclosure. Althoughprocess 3600 is illustrated as a sequence, it is to be understood thattwo or more of the steps may be executed concurrently, and the entireprocess may be executed in real-time, as discussed previously herein.

Process 3600 may begin with step 3602. At step 3602, the at least oneprocessor may receive a real time video feed from at least one imagesensor (e.g., image sensor 226) associated with a mobile device (e.g.,mobile communications device 115), consistent with some embodiments ofthe present disclosure. Then, at step 3604, the at least one processormay receive image-based information associated with at least onepreviously captured image of a wound (e.g., wounds 3400 and 3500),consistent with some embodiments of the present disclosure. For example,the at least one processor may access at least one data structure (e.g.,database 146) to retrieve data associated with the at least one capturedimage.

After the image-based information has been received, process 3600 mayproceed to step 3606. At step 3606, the at least one processor maygenerate, using the video feed and the image-based information, anoverlay (e.g., overlays 3420 and 3520) including at least one indication(e.g., elements 3422, 3424, 3500A, 3522A, 3522B, 3524A, and 3524B) of acondition of the wound in the at least one previously captured image,consistent with some embodiments of the present disclosure. Process 3600may end at step 3608. At step 3608, the at least one processor maydisplay the overlay on at least one user interface (e.g., touch screen218), consistent with some embodiments of the present disclosure. The atleast one user interface may be configured to display the overlay in aposition associated with (e.g., superimposed on) a position of the woundin the video feed.

The foregoing description has been presented for purposes ofillustration. It is not exhaustive and is not limited to the preciseforms or embodiments disclosed. Modifications and adaptations will beapparent to those skilled in the art from consideration of thespecification and practice of the disclosed embodiments. Additionally,although aspects of the disclosed embodiments are described as beingstored in memory, one skilled in the art will appreciate that theseaspects can also be stored on other types of computer readable media,such as secondary storage devices, e.g., hard disks or CD ROM, or otherforms of RAM or ROM, USB media, DVD, Blu-ray, Ultra HD Blu-ray, or otheroptical drive media.

Computer programs based on the written description and disclosed methodsare within the skills of an experienced developer. The various programsor program modules can be created using any of the techniques known toone skilled in the art or can be designed in connection with existingsoftware. For example, program sections or program modules can bedesigned in or by means of .Net Framework, .Net Compact Framework (andrelated languages, such as Visual Basic, C, etc.), Java, C++,Objective-C, HTML, HTML/AJAX combinations, XML, or HTML with includedJava applets.

Moreover, while illustrative embodiments have been described herein, thescope of any and all embodiments having equivalent elements,modifications, omissions, combinations (e.g., of aspects across variousembodiments), adaptations and/or alterations as would be appreciated bythose skilled in the art based on the present disclosure. The examplesare to be construed as non-exclusive. Furthermore, the steps of thedisclosed methods may be modified in any manner, including by reorderingsteps and/or inserting or deleting steps. It is intended, therefore,that the specification and examples be considered as illustrative only.

What is claimed is:
 1. A non-transitory computer readable medium storingdata and computer implementable instructions that, when executed by atleast one processor, cause the at least one processor to performoperations for displaying an overlay on wounds, the operationscomprising: receiving a real time video feed; receiving image-basedinformation associated with at least one previously captured image of awound; generating, using the video feed and the image-based information,an overlay including an indication of a condition of the wound in the atleast one previously captured image; and displaying, on at least oneuser interface, the overlay, wherein the at least one user interface isconfigured to display the overlay in a position associated with aposition of the wound in the video feed.
 2. The non-transitory computerreadable medium of claim Error! Reference source not found., wherein theat least one previously captured image is captured at least one daybefore the video feed is captured.
 3. The non-transitory computerreadable medium of claim Error! Reference source not found., wherein thevideo feed includes a plurality of wounds, and the operations furthercomprise selecting the wound from the plurality of wounds.
 4. Thenon-transitory computer readable medium of claim Error! Reference sourcenot found., wherein receiving the image-based information comprises:accessing a plurality of records, each record of the plurality ofrecords corresponding to a different wound; selecting a recordcorresponding to the wound of the plurality of records based on thevideo feed; and obtaining the image-based information from the selectedrecord.
 5. The non-transitory computer readable medium of claim Error!Reference source not found., wherein the overlay is displayed on theuser interface feed in real time.
 6. The non-transitory computerreadable medium of claim Error! Reference source not found., wherein theoverlay is displayed in real time using a transparent optical systemincluded in a wearable device, the real time video feed being a realtime video feed captured using an image sensor included in the wearabledevice, the wound being visible to a user wearing the wearable devicethrough the transparent optical system, and the display of the overlaybeing configured to make the overlay appear to the user wearing thewearable device at least partly over the wound.
 7. The non-transitorycomputer readable medium of claim Error! Reference source not found.,wherein the at least one user interface is associated with a mobiledevice.
 8. The non-transitory computer readable medium of claim 7,wherein the at least one user interface is configured to automaticallyadjust the position of the displayed overlay based on detected movementof the mobile device.
 9. The non-transitory computer readable medium ofclaim Error! Reference source not found., wherein the operations furthercomprise: receiving second image-based information associated with asecond at least one previously captured image of the wound; andincluding a second indication in the overlay, the second indicationbeing an indication of a condition of the wound in the second at leastone previously captured image, the condition of the wound in the secondat least one previously captured image differing from the condition ofthe wound in the at least one previously captured image.
 10. Thenon-transitory computer readable medium of claim 8, wherein the overlayfurther includes an indication of a capturing time associated with theat least one previously captured image and an indication of a capturingtime associated with the second at least one previously captured image.11. The non-transitory computer readable medium of claim 8, wherein thecondition of the wound in the at least one previously captured imagecorresponds to a first point in time, the condition of the wound in thesecond at least one previously captured image corresponds to a secondpoint in time, and the operations further comprise: using theimage-based information and the second image-based information todetermine a condition of the wound corresponding to a third point intime, the third point in time differing from the first point in time andthe second point in time; and including a third indication in theoverlay, the third indication being an indication of a condition of thewound corresponding to the third point in time.
 12. The non-transitorycomputer readable medium of claim Error! Reference source not found.,wherein the operations further comprise: calculating a convolution of atleast part of the at least one previously captured image to derive aresult value; in response to a first result value, including in theoverlay a first indication of the condition of the wound in the at leastone previously captured image; and in response to a second result value,including in the overlay a second indication of the condition of thewound in the at least one previously captured image, the secondindication differing from the first indication.
 13. The non-transitorycomputer readable medium of claim Error! Reference source not found.,wherein the indication of the condition of the wound includes a visualindication of a contour of the wound in the at least one previouslycaptured image.
 14. The non-transitory computer readable medium of claimError! Reference source not found., wherein the indication of thecondition of the wound includes an indication of at least onemeasurement of the wound in the at least one previously captured image.15. The non-transitory computer readable medium of claim 14, wherein theat least one measurement includes at least one of a length, an area, avolume, or a depth of the wound.
 16. The non-transitory computerreadable medium of claim Error! Reference source not found., wherein theindication of the condition of the wound includes a visual indication ofa segment of the wound in the at least one previously captured imagecorresponding to a tissue type.
 17. The non-transitory computer readablemedium of claim Error! Reference source not found., wherein theindication of the condition of the wound includes a visual indication ofa color of a portion of the wound in the at least one previouslycaptured image.
 18. The non-transitory computer readable medium of claimError! Reference source not found., wherein the indication of thecondition of the wound includes a visual indication of a severity of thewound in the at least one previously captured image.
 19. A system fordisplaying an overlay on a wound, the system comprising at least oneprocessing unit configured to: receive a real time video feed; receiveimage-based information associated with at least one previously capturedimage of a wound; generate, using the video feed and the image-basedinformation, an overlay including an indication of a condition of thewound in the at least one previously captured image; and display, on atleast one user interface, the overlay, wherein the at least one userinterface is configured to display the overlay in a position associatedwith a position of the wound in the video feed.
 20. A method fordisplaying an overlay on a wound in, the method comprising: receiving areal time video feed; receiving image-based information associated withat least one previously captured image of a wound; generating, using thevideo feed and the image-based information, an overlay including anindication of a condition of the wound in the at least one previouslycaptured image; and displaying, on at least one user interface, theoverlay, wherein the at least one user interface is configured todisplay the overlay in a position associated with a position of thewound in the video feed.